mlsanityData to 7 Oct 2026

Mapping Social Choice Theory to RLHF

arXiv 2404.13038 ↗First posted 19 Apr 2024cs.AI cs.CYcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent work on the limitations of using reinforcement learning from human feedback (RLHF) to incorporate human preferences into model behavior often raises social choice theory as a reference point. Social choice theory's analysis of settings such as voting mechanisms provides technical infrastructure that can inform how to aggregate human preferences amid disagreement. We analyze the problem settings of social choice and RLHF, identify key differences between them, and discuss how these differences may affect the RLHF interpretation of well-known technical results in social choice.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 30 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

20 of 30 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  2. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  3. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan

Other references · 13

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  2. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  3. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  4. 2023Generative Social Choicecited by 43
  5. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 35
  6. 2023Algorithmic Collective Action in Machine Learningcited by 30
  7. 2021Learning Reward Functions from Scale Feedbackcited by 24
  8. 2016Learning Mixtures of Plackett-Luce Modelscited by 19
  9. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 17
  10. 2021Distortion in Social Choice Problems: The First 15 Years and Beyondcited by 16
  11. 2018Efficiently Learning Mixtures of Mallows Modelscited by 13
  12. 2022Representation with Incomplete Votescited by 13
1 more
  1. 2015Sample Complexity for Winner Prediction in Electionscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 9.

26 papers.

Sort:

By area:

  1. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  2. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  3. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  4. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  5. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  6. 2025Jackpot! Alignment as a Maximal Lotterycited by 17
  7. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  8. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matchingcited by 10
  9. 2025Clone-Robust AI Alignmentcited by 7
  10. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 6
  11. 2024Learning How to Vote with Principles: Axiomatic Insights Into the Collective Decisions of Neural Networkscited by 5
  12. 2024Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignmentcited by 4
14 more
  1. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 3
  2. 2026Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracycited by 2
  3. 2025Adaptive Preference Aggregationcited by 1
  4. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  5. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modelingcited by 1
  6. 2025Maximizing the efficiency of human feedback in AI alignment: a comparative analysiscited by 1
  7. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  8. 2026Power and Limitations of Aggregation in Compound AI Systemscited by 1
  9. 2026Open Problems in Differentiable Social Choice: Learning Mechanisms, Decisions, and Alignmentcited by 0
  10. 2026When to Ask a Question: Understanding Communication Strategies in Generative AI Toolscited by 0
  11. 2026The Representation-Rationalizability Tradeoff in Reward Learningcited by 0
  12. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  13. 2026AI Alignment From Social Choice Perspectivescited by 0
  14. 2026Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Influencing Humans to Conform to Preference Models for RLHF
  2. 2025A Descriptive and Normative Theory of Human Beliefs in RLHF
  3. 2023A Survey of Reinforcement Learning from Human Feedback
  4. 2023Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling
  8. 2024Representative Social Choice: From Learning Theory to AI Alignment