mlsanityData to 14 Oct 2026

AI Alignment From Social Choice Perspectives

arXiv 2606.21550 ↗First posted 19 Jun 2026cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Alignment from human feedback uses human judgments about model outputs to steer the behavior of language models after pretraining. When those judgments reflect conflicting views of desirable behavior, the learned objective becomes an aggregate determination of what the model should prefer. We survey recent work that has studied this aggregation problem through the lens of social choice theory. We illustrate how the social choice perspective helps identify failure modes in the feedback aggregation layer and reveals a broader design space for handling disagreement in explicit and principled ways.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 92 of its 142 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

92 of 142 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modeling20 Apr
  2. 2025Evaluating the Diversity and Quality of LLM Generated Content16 Apr
  3. 2025Value Profiles for Encoding Human Variation19 Mar
  4. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?18 Oct
  5. 2024Beyond Preferences in AI Alignment30 Aug
  6. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration22 Jun
  7. 2024Collective Constitutional AI: Aligning a Language Model with Public Input12 Jun
  8. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
  9. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback16 Apr
  10. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications24 Oct
  11. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  12. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov

Other references · 80

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2022Red Teaming Language Models with Language Modelscited by 1,026
  8. 2023Whose Opinions Do Language Models Reflect?cited by 773
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  12. 2020Artificial Intelligence, Values and Alignmentcited by 448
38 more
  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  3. 2019Measurement and Fairnesscited by 308
  4. 2021Unsolved Problems in ML Safetycited by 292
  5. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  7. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  8. 2023Nash Learning from Human Feedbackcited by 224
  9. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  10. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  11. 2021Alignment of Language Agentscited by 168
  12. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  13. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  14. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  15. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  16. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  17. 2020A Distributional Approach to Controlled Text Generationcited by 119
  18. 2015Contextual Dueling Banditscited by 101
  19. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  20. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  21. 2024How to Evaluate Reward Models for RLHFcited by 88
  22. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  23. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  24. 2022Models of human preference for learning reward functionscited by 54
  25. 2024Axioms for AI Alignment from Human Feedbackcited by 52
  26. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  27. 2023Generative Social Choicecited by 43
  28. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  29. 2021The Challenge of Value Alignment: from Fairer Algorithms to AI Safetycited by 35
  30. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  31. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  32. 2023Aligning Language Models with Human Preferences via a Bayesian Approachcited by 27
  33. 2024Mapping Social Choice Theory to RLHFcited by 27
  34. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  35. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  36. 2026How RLHF Amplifies Sycophancycited by 23
  37. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 22
  38. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Context-dependent agent evaluation with orthogonal equilibrium learningcited by 0
  2. 2026Axiom Satisfiability of Linear Rewards in Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment
  2. 2026Align AI to Dynamic Human-AI Workflows
  3. 2026Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction
  4. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  5. 2026Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignment
  6. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  7. 2024Representative Social Choice: From Learning Theory to AI Alignment
  8. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoning