mlsanityData to 13 Oct 2026

Toward a Theory of Value in AI Alignment

arXiv 2608.10327 ↗First posted 10 Aug 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Can AI systems be aligned to human values? The popularization of large language models (LLMs) and multi-modal foundation models has seen a rise in harms spanning from toxic speech and hallucinations to AI agents executing unauthorized actions. Within the field of AI safety, these harmful instances are often framed as the alignment problem, or of models being misaligned with human values. Researchers have responded by pursuing applied and theoretical AI value alignment efforts, often without specifying what they mean by human values. How does the field of AI value alignment conceive of human values? How are these conceptions of values technically operationalized and evaluated? What does the emergent theory of value from this field signify for the future of AI? We annotated 94 value alignment research papers to discern their implicit theory of values in AI. The majority do not define values, relying heavily on preferences as a stand in that runs the risk of reducing complex culturally situated concepts down to binary choices. As researchers dispense with using human annotators for model training and evaluation, turning instead to synthetic data and autorater approaches to aligning and evaluating models, we identify the potential to close off alternative methods for contesting and enacting values in foundation models. In making AI value alignments philosophical commitments explicit, we seek to bring great specificity and under explored perspectives in the debate on whether and how AI can address human values.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 105 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

46 of 105 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Societal Alignment Frameworks Can Improve LLM Alignment27 Feb
  2. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback24 Oct
  3. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations26 Jun
  4. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation8 Feb
  5. 2023Constructive Large Language Models Alignment with Diverse Feedback10 Oct
  6. 2023Training Socially Aligned Language Models on Simulated Social Interactions26 May
  7. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar
  8. 2021Value alignment: a formal approach18 Oct

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  3. 2017Deep reinforcement learning from human preferencescited by 3,871
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,041
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 717
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 448
  8. 2018Scalable agent alignment via reward modeling: a research directioncited by 443
  9. 2020Artificial Intelligence, Values and Alignmentcited by 443
  10. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 428
  11. 2016Cooperative Inverse Reinforcement Learningcited by 299
  12. 2023AI Alignment: A Comprehensive Surveycited by 261
26 more
  1. 2023Large Language Model Alignment: A Surveycited by 243
  2. 2021The Values Encoded in Machine Learning Researchcited by 172
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 152
  4. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 130
  5. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  7. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  8. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 66
  9. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 59
  10. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMscited by 54
  11. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  12. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  13. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  14. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  15. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samplescited by 25
  16. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Studycited by 23
  17. 2024Two Types of AI Existential Risk: Decisive and Accumulativecited by 22
  18. 2023Concrete Problems in AI Safety, Revisitedcited by 22
  19. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modelingcited by 21
  20. 2022A Human Rights-Based Approach to Responsible AIcited by 20
  21. 2024HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Routercited by 16
  22. 2024Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHFcited by 15
  23. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  24. 2024Reward Learning From Preference With Tiescited by 14
  25. 2026Toward Comprehensive Risk Assessments and Assurance of AI-Based Systemscited by 4
  26. 2024Do LLMs Have the Generalization Ability in Conducting Causal Inference?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs
  2. 2023A Multi-Level Framework for the AI Alignment Problem
  3. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives
  4. 2024Position: Towards Bidirectional Human-AI Alignment
  5. 2025Cultural Bias in Large Language Models: Evaluating AI Agents through Moral Questionnaires
  6. 2025Evaluating AI Alignment in LLMs: Output Analysis of Value Priorities Across 75 Models with Human Benchmarking
  7. 2024The AI Alignment Paradox
  8. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Models