# AI Alignment and Social Choice: Fundamental Limitations and Policy Implications

arXiv: https://arxiv.org/abs/2310.16048 · HTML: https://mlsanity.com/paper/2310.16048/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 12 of 12)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Artificial Intelligence: the global landscape of ethics guidelines](https://mlsanity.com/paper/1906.11668/) (2019)
- [A Voting-Based System for Ethical Decision Making](https://mlsanity.com/paper/1709.06692/) (2017)

## Built on by (most cited first, 30 of 36)

- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF](https://mlsanity.com/paper/2312.08358/) (2023)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2504.12501/) (2025)
- [Axioms for AI Alignment from Human Feedback](https://mlsanity.com/paper/2405.14758/) (2024)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking](https://mlsanity.com/paper/2409.15268/) (2024)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [Mapping Social Choice Theory to RLHF](https://mlsanity.com/paper/2404.13038/) (2024)
- [Improving Context-Aware Preference Modeling for Language Models](https://mlsanity.com/paper/2407.14916/) (2024)
- [Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?](https://mlsanity.com/paper/2505.23749/) (2025)
- [Virtual Agent Economies](https://mlsanity.com/paper/2509.10147/) (2025)
- [Jackpot! Alignment as a Maximal Lottery](https://mlsanity.com/paper/2501.19266/) (2025)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [Pairwise Calibrated Rewards for Pluralistic Alignment](https://mlsanity.com/paper/2506.06298/) (2025)
- [Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching](https://mlsanity.com/paper/2505.20627/) (2025)
- [Representative Social Choice: From Learning Theory to AI Alignment](https://mlsanity.com/paper/2410.23953/) (2024)
- [Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory](https://mlsanity.com/paper/2506.12350/) (2025)
- [Adaptive Alignment: Dynamic Preference Adjustments via Multi-Objective Reinforcement Learning for Pluralistic AI](https://mlsanity.com/paper/2410.23630/) (2024)
- [Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework](https://mlsanity.com/paper/2506.05619/) (2025)
- [The Right to AI](https://mlsanity.com/paper/2501.17899/) (2025)
- [From Logic Monopoly to Social Contract: Separation of Power and the Institutional Foundations for Autonomous Agent Economies](https://mlsanity.com/paper/2603.25100/) (2026)
- [AI Alignment From Social Choice Perspectives](https://mlsanity.com/paper/2606.21550/) (2026)
- [Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies](https://mlsanity.com/paper/2412.00033/) (2024)
- [Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling](https://mlsanity.com/paper/2510.23631/) (2025)
- [Power and Limitations of Aggregation in Compound AI Systems](https://mlsanity.com/paper/2602.21556/) (2026)
- [Curriculum Group Policy Optimization: Adaptive Sampling for Unleashing the Potential of Text-to-Image Generation](https://mlsanity.com/paper/2605.17807/) (2026)
- [Embracing Dialectic Intersubjectivity: Coordination of Different Perspectives in Content Analysis with LLM Persona Simulation](https://mlsanity.com/paper/2502.00903/) (2025)
- [Balancing Innovation and Integrity: AI Integration in Liberal Arts College Administration](https://mlsanity.com/paper/2503.05747/) (2025)

## Abstract (arXiv)

Aligning AI agents to human intentions and values is a key bottleneck in building safe and deployable AI applications. But whose values should AI agents be aligned with? Reinforcement learning with human feedback (RLHF) has emerged as the key framework for AI alignment. RLHF uses feedback from human reinforcers to fine-tune outputs; all widely deployed large language models (LLMs) use RLHF to align their outputs to human values. It is critical to understand the limitations of RLHF and consider policy challenges arising from these limitations. In this paper, we investigate a specific challenge in building RLHF systems that respect democratic norms. Building on impossibility results in social choice theory, we show that, under fairly broad assumptions, there is no unique voting protocol to universally align AI systems using RLHF through democratic processes. Further, we show that aligning AI agents with the values of all individuals will always violate certain private ethical preferences of an individual user i.e., universal AI alignment using RLHF is impossible. We discuss policy implications for the governance of AI systems built using RLHF: first, the need for mandating transparent voting rules to hold model builders accountable. Second, the need for model builders to focus on developing AI agents that are narrowly aligned to specific user groups.