Position: Align AI to Our Aspirations, Not Our Flaws
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
We argue that aligning AI to aggregated human preferences is the wrong target. With current technology, one can train AIs to share the values of a Silicon Valley techno-optimist, a degrowth environmentalist, a national-conservative culture warrior, a single-party state cadre, or a devout religious traditionalist. We should not. Human values produce societies that thrive or fail on the merits of those values - from failed states and extreme inequality to declining happiness, political polarization, and government dysfunction in the world's wealthiest democracies. The pluralistic-alignment program correctly diagnoses that there is no single "humanity" to align with, but is dangerous if taken as the main directive. We argue that AI should be trained to a non-negotiable floor of objective alignment goals - competence, bounded by the constraints of factual accuracy, honesty, and lawfulness and that pluralism belongs at the surface (language, register, conventions, missing-context defaults) and across the wide band of legitimate value tradeoffs that respect the floor, but not at the level of values that violate it. We highlight the empirical reality of unfiltered pluralistic values, propose four commitments as a constructive alternative, and engage six credible objections: commercial pressure and practical feasibility, democratic legitimacy, regulatory compliance, over-reliance on institutionalist explanations, the charge that the floor itself is culturally laden, and the limits of Coherent Extrapolated Volition.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 131 references that are in the map. First posted on arXiv 11 Jun 2026.
Builds on
25 of 131 references.
Closest ancestors · 7
Same direction, newest first
- 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences7 Oct
- 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization8 Mar
- 2024Beyond Preferences in AI Alignment30 Aug
- 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback16 Apr
- 2024A Roadmap to Pluralistic Alignment7 Feb
- 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
- 2022In conversation with Artificial Intelligence: aligning language models with human values1 Sep
Other references · 18
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023Let's Verify Step by Stepcited by 3,598
- 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
- 2020Artificial Intelligence, Values and Alignmentcited by 443
- 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
- 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
6 more
- 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
- 2023From Values to Opinions: Predicting Human Behaviors and Stances Using Value-Injected Large Language Modelscited by 10
- 2024Democratic AI is Possible. The Democracy Levels Framework Shows How It Might Workcited by 8
- 2025Biases Propagate in Encoder-based Vision-Language Models: A Systematic Analysis From Intrinsic Measures to Zero-shot Retrieval Outcomescited by 6
- 2026PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harmcited by 5
- 2026"Label from Somewhere": Reflexive Annotating for Situated AI Alignmentcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026A Roadmap to Impactful Pluralistic Alignment Researchcited by 0
- 2026A Roadmap to Impactful Pluralistic Alignment Research24 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignment
- 2026From Sycophantic Consensus to Pluralistic Repair: Why AI Alignment Must Surface Disagreement
- 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications
- 2023A Multi-Level Framework for the AI Alignment Problem
- 2024The AI Alignment Paradox
- 2021The Challenge of Value Alignment: from Fairer Algorithms to AI Safety
- 2026Toward a Theory of Value in AI Alignment
- 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs