# Mapping Social Choice Theory to RLHF

arXiv: https://arxiv.org/abs/2404.13038 · HTML: https://mlsanity.com/paper/2404.13038/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 20 of 20)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [Statistical Rejection Sampling Improves Preference Optimization](https://mlsanity.com/paper/2309.06657/) (2023)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons](https://mlsanity.com/paper/2301.11270/) (2023)
- [Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF](https://mlsanity.com/paper/2312.08358/) (2023)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [Generative Social Choice](https://mlsanity.com/paper/2309.01291/) (2023)
- [The History and Risks of Reinforcement Learning and Human Feedback](https://mlsanity.com/paper/2310.13595/) (2023)
- [AI Alignment and Social Choice: Fundamental Limitations and Policy Implications](https://mlsanity.com/paper/2310.16048/) (2023)
- [Algorithmic Collective Action in Machine Learning](https://mlsanity.com/paper/2302.04262/) (2023)
- [Learning Reward Functions from Scale Feedback](https://mlsanity.com/paper/2110.00284/) (2021)
- [Learning Mixtures of Plackett-Luce Models](https://mlsanity.com/paper/1603.07323/) (2016)
- [Incorporating Worker Perspectives into MTurk Annotation Practices for NLP](https://mlsanity.com/paper/2311.02802/) (2023)
- [Distortion in Social Choice Problems: The First 15 Years and Beyond](https://mlsanity.com/paper/2103.00911/) (2021)
- [Efficiently Learning Mixtures of Mallows Models](https://mlsanity.com/paper/1808.05731/) (2018)
- [Representation with Incomplete Votes](https://mlsanity.com/paper/2211.15608/) (2022)
- [Sample Complexity for Winner Prediction in Elections](https://mlsanity.com/paper/1502.04354/) (2015)

## Built on by (most cited first, 27 of 27)

- [Axioms for AI Alignment from Human Feedback](https://mlsanity.com/paper/2405.14758/) (2024)
- [Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences](https://mlsanity.com/paper/2405.15065/) (2024)
- [Improving Context-Aware Preference Modeling for Language Models](https://mlsanity.com/paper/2407.14916/) (2024)
- [Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?](https://mlsanity.com/paper/2505.23749/) (2025)
- [Direct Alignment with Heterogeneous Preferences](https://mlsanity.com/paper/2502.16320/) (2025)
- [Jackpot! Alignment as a Maximal Lottery](https://mlsanity.com/paper/2501.19266/) (2025)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching](https://mlsanity.com/paper/2505.20627/) (2025)
- [Clone-Robust AI Alignment](https://mlsanity.com/paper/2501.09254/) (2025)
- [Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory](https://mlsanity.com/paper/2506.12350/) (2025)
- [Learning How to Vote with Principles: Axiomatic Insights Into the Collective Decisions of Neural Networks](https://mlsanity.com/paper/2410.16170/) (2024)
- [Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignment](https://mlsanity.com/paper/2410.11221/) (2024)
- [Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework](https://mlsanity.com/paper/2506.05619/) (2025)
- [Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracy](https://mlsanity.com/paper/2605.01642/) (2026)
- [AI Alignment From Social Choice Perspectives](https://mlsanity.com/paper/2606.21550/) (2026)
- [Adaptive Preference Aggregation](https://mlsanity.com/paper/2503.10215/) (2025)
- [Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences](https://mlsanity.com/paper/2510.15716/) (2025)
- [Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling](https://mlsanity.com/paper/2510.23631/) (2025)
- [Maximizing the efficiency of human feedback in AI alignment: a comparative analysis](https://mlsanity.com/paper/2511.12796/) (2025)
- [Robust AI Evaluation through Maximal Lotteries](https://mlsanity.com/paper/2602.21297/) (2026)
- [Power and Limitations of Aggregation in Compound AI Systems](https://mlsanity.com/paper/2602.21556/) (2026)
- [The Representation-Rationalizability Tradeoff in Reward Learning](https://mlsanity.com/paper/2606.00291/) (2026)
- [Open Problems in Differentiable Social Choice: Learning Mechanisms, Decisions, and Alignment](https://mlsanity.com/paper/2602.03003/) (2026)
- [When to Ask a Question: Understanding Communication Strategies in Generative AI Tools](https://mlsanity.com/paper/2605.11240/) (2026)
- [Efficient Exploration for Iterative Nash Preference Optimization](https://mlsanity.com/paper/2606.01382/) (2026)
- [Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignment](https://mlsanity.com/paper/2608.24046/) (2026)
- [Axiom Satisfiability of Linear Rewards in Alignment](https://mlsanity.com/paper/2610.06892/) (2026)

## Abstract (arXiv)

Recent work on the limitations of using reinforcement learning from human feedback (RLHF) to incorporate human preferences into model behavior often raises social choice theory as a reference point. Social choice theory's analysis of settings such as voting mechanisms provides technical infrastructure that can inform how to aggregate human preferences amid disagreement. We analyze the problem settings of social choice and RLHF, identify key differences between them, and discuss how these differences may affect the RLHF interpretation of well-known technical results in social choice.