July 29th, 2026 ยท Tags: AI alignment, value alignment, AI safety, RLHF
The central promise of modern AI is that it will do what we want. But buried under every chatbot and recommender system, every autonomous agent is a question nobody has solved: what does "what we want" actually mean?
The Preference Trap #
The dominant approach to alignment, Reinforcement Learning from Human Feedback (RLHF), works by collecting human preferences (which of two responses is better) and training a reward model to predict those judgments. The problem isn't that this fails in practice. It's that it can't succeed in principle.
A formal result called the Alignment Trilemma proves that no RLHF system can simultaneously capture the full diversity of human values, remain computationally tractable, and resist adversarial manipulation. Current systems resolve this by simply not representing diverse values. They amplify majority viewpoints and suppress minority perspectives as if they were measurement noise.
The scale of the suppression is measurable. A 2025 study of feedback aggregation found 79% of prompts had multiple valid responses that single-winner methods discarded (Hidden Consensus: Preference-Validity Compression in Human Feedback).
There Are No Correct Weights #
A deeper problem goes beyond engineering. Human values are what philosophers call plural and incommensurable. Safety and freedom pull in opposite directions. Honesty and kindness conflict. There is no mathematically correct weighting of these values. There never will be.
A paper on the "specification trap" argues that any static value encoding (reward function, constitution, utility function) is structurally incapable of producing robust alignment. The trap activates the moment you close the specification and stop updating it. Values shift over time. Contexts change. The AI itself creates new situations its training never covered (The Specification Trap).
The Political Problem Nobody Talks About #
Even if we could solve the technical side, we'd still face a political one. When AI systems make morally consequential decisions, who gets to decide which values they encode? The developers? A democratic vote? The most vocal users?
Constitutional AI and RLHF both claim some form of democratic legitimacy, but neither actually delivers it. Training processes are opaque. Stakeholders can't understand how principles become algorithms. People who disagree with a system's outputs have no recourse and no reason to accept them. The leading alignment methods all fail to accommodate reasonable moral disagreement, which is a problem when these systems are deployed at society scale (Moral Disagreement and the Limits of AI Value Alignment).
What's Actually Being Done #
A few research directions are worth watching. Pluralistic alignment aims to build systems that represent multiple valid perspectives rather than collapsing them into one. Deliberative alignment trains explicit normative reasoning instead of behavioral dispositions. Open specification keeps value representations responsive to ongoing human input rather than freezing them at training time.
None of these are in production yet. The frontier labs have not adopted pluralism as a goal, and the evaluation infrastructure doesn't exist to measure it (A Roadmap to Impactful Pluralistic Alignment Research).
Why This Matters #
The alignment problem isn't a bug that engineering will fix. It's a feature of the fact that human values are contested, plural, and alive. The systems we build today encode a frozen snapshot of someone's idea of right and wrong. The question is whose.