mlsanityData to 14 Oct 2026

Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory

arXiv 2506.12350 ↗First posted 14 Jun 2025stat.ML cs.AI cs.LGcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Despite its empirical success, Reinforcement Learning from Human Feedback (RLHF) has been shown to violate almost all the fundamental axioms in social choice theory -- such as majority consistency, pairwise majority consistency, and Condorcet consistency. This raises a foundational question: why does RLHF perform so well in practice if it fails these seemingly essential properties? In this paper, we resolve this paradox by showing that under mild and empirically plausible assumptions on the preference profile, RLHF does satisfy pairwise majority and Condorcet consistency. These assumptions are frequently satisfied in real-world alignment tasks, offering a theoretical explanation for RLHF's strong practical performance. Furthermore, we show that a slight modification to the reward modeling objective can ensure pairwise majority or Condorcet consistency even under general preference profiles, thereby improving the alignment process. Finally, we go beyond classical axioms in economic and social choice theory and introduce new alignment criteria -- preference matching, preference equivalence, and group preference matching -- that better reflect the goal of learning distributions over responses. We show that while RLHF satisfies the first two properties, it fails to satisfy the third. We conclude by discussing how future alignment methods may be designed to satisfy all three.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 55 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

40 of 55 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching27 May
  2. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium14 Mar
  3. 2025Jackpot! Alignment as a Maximal Lottery31 Jan
  4. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment22 Oct
  5. 2024Axioms for AI Alignment from Human Feedback23 May
  6. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation30 Apr
  7. 2024Mapping Social Choice Theory to RLHF19 Apr
  8. 2024Dataset Reset Policy Optimization for RLHF12 Apr
  9. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback8 Mar
  10. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  11. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  12. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
6 more
  1. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Data17 Dec
  2. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  3. 2023Nash Learning from Human Feedback1 Dec
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  9. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
10 more
  1. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  2. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  3. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  4. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  5. 2024Learn Your Reference Model for Real Good Alignmentcited by 57
  6. 2024Asymptotics of Language Model Alignmentcited by 47
  7. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 36
  8. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  9. 2024Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmeticcited by 14
  10. 2024Learning Linear Utility Functions From Pairwise Comparison Queriescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamicscited by 2
  2. 2026AI Alignment From Social Choice Perspectivescited by 2
  3. 2026The Representation-Rationalizability Tradeoff in Reward Learningcited by 1
  4. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scalingcited by 0
  5. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0
  6. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0
  7. 2026Axiom Satisfiability of Linear Rewards in Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2025Influencing Humans to Conform to Preference Models for RLHF
  3. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences
  4. 2023The History and Risks of Reinforcement Learning and Human Feedback
  5. 2026Procedural Fairness Failures in RLHF from Preference Averaging
  6. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences