mlsanityData to 14 Oct 2026

Axioms for AI Alignment from Human Feedback

arXiv 2405.14758 ↗First posted 23 May 2024cs.GT cs.AI cs.LGcited by 52

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In the context of reinforcement learning from human feedback (RLHF), the reward function is generally derived from maximum likelihood estimation of a random utility model based on pairwise comparisons made by humans. The problem of learning a reward function is one of preference aggregation that, we argue, largely falls within the scope of social choice theory. From this perspective, we can evaluate different aggregation methods via established axioms, examining whether these methods meet or fail well-known standards. We demonstrate that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms. In response, we develop novel rules for learning reward functions with strong axiomatic guarantees. A key innovation from the standpoint of social choice is that our problem has a linear structure, which greatly restricts the space of feasible rules and leads to a new paradigm that we call linear social choice.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 32 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

15 of 32 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation30 Apr
  2. 2024Mapping Social Choice Theory to RLHF19 Apr
  3. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback8 Mar
  4. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  5. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  7. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 6

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2017A Voting-Based System for Ethical Decision Makingcited by 55
  4. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 36
  5. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  6. 2024Learning Linear Utility Functions From Pairwise Comparison Queriescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 24; 2026: 24.

52 papers.

Sort:

By area:

  1. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  2. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  3. 2026How RLHF Amplifies Sycophancycited by 23
  4. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 22
  5. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  6. 2025Jackpot! Alignment as a Maximal Lotterycited by 17
  7. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  8. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 14
  9. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHFcited by 11
  10. 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
  11. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matchingcited by 10
  12. 2024Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewardscited by 9
38 more
  1. 2025Clone-Robust AI Alignmentcited by 9
  2. 2024Representative Social Choice: From Learning Theory to AI Alignmentcited by 8
  3. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 7
  4. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  5. 2024SEAL: Systematic Error Analysis for Value ALignmentcited by 4
  6. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 4
  7. 2025Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignmentcited by 3
  8. 2025Judging with Confidence: Calibrating Autoraters to Preference Distributionscited by 3
  9. 2026Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theoremscited by 3
  10. 2025Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Modelscited by 2
  11. 2025Welfare as a Guiding Principle for Machine Learning -- From Compass, to Lens, to Roadmapcited by 2
  12. 2025Optimized Distortion in Linear Social Choicecited by 2
  13. 2025The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneitycited by 2
  14. 2025The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curationcited by 2
  15. 2026Differential Voting: Loss Functions For Axiomatically Diverse Aggregation of Heterogeneous Preferencescited by 2
  16. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 2
  17. 2026Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracycited by 2
  18. 2026AI Alignment From Social Choice Perspectivescited by 2
  19. 2025Adaptive Preference Aggregationcited by 1
  20. 2025The Alignment Bottleneckcited by 1
  21. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modelingcited by 1
  22. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  23. 2026Finding Common Ground in a Sea of Alternativescited by 1
  24. 2026Linear Social Choice with Few Queries: A Moment-Based Approachcited by 1
  25. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 1
  26. 2026The End Justifies the Mean: A Linear Ranking Rule for Proportional Sequential Decisionscited by 1
  27. 2026The Representation-Rationalizability Tradeoff in Reward Learningcited by 1
  28. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Languagecited by 1
  29. 2026Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Alignercited by 1
  30. 2025Axiomatic Choicecited by 0
  31. 2026Open Problems in Differentiable Social Choice: Learning Mechanisms, Decisions, and Alignmentcited by 0
  32. 2026When to Ask a Question: Understanding Communication Strategies in Generative AI Toolscited by 0
  33. 2026AI of the People, by the People, for the People: A Social Choice Approach to Collective Control of Artificial Intelligencecited by 0
  34. 2026Internal Pluralism and the Limits of Pairwise Comparisonscited by 0
  35. 2026Group Preference Collapse in Personalized Multimodal Large Language Modelscited by 0
  36. 2026More Data, Worse Decisions? Preference Reversals in Neural Networks under Gram Incompatibilitycited by 0
  37. 2026Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignmentcited by 0
  38. 2026Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignmentcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Models of human preference for learning reward functions
  2. 2025Influencing Humans to Conform to Preference Models for RLHF
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  6. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  7. 2023Learning Optimal Advantage from Preferences and Mistaking it for Reward
  8. 2025Maximizing the efficiency of human feedback in AI alignment: a comparative analysis