mlsanityData to 7 Oct 2026

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

arXiv 2404.10271 ↗First posted 16 Apr 2024cs.LG cs.AI cs.CLcited by 95

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise problematic behavior, such as helping to commit crimes or producing racist text. One approach to fine-tuning, called reinforcement learning from human feedback, learns from humans' expressed preferences over multiple outputs. Another approach is constitutional AI, in which the input from humans is a list of high-level principles. But how do we deal with potentially diverging input from humans? How can we aggregate the input into consistent data about "collective" preferences or otherwise use it to make collective choices about model behavior? In this paper, we argue that the field of social choice is well positioned to address these questions, and we discuss ways forward for this agenda, drawing on discussions in a recent workshop on Social Choice for AI Ethics and Safety held in Berkeley, CA, USA in December 2023.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 108 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

45 of 108 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications24 Oct
  2. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2020Artificial Intelligence, Values and Alignmentcited by 443
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  11. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  12. 2023Preference Ranking Optimization for Human Alignmentcited by 283
28 more
  1. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  2. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  3. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  4. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  5. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  6. 2023Eliciting Human Preferences with Language Modelscited by 90
  7. 2024The Political Preferences of LLMscited by 83
  8. 2017A Voting-Based System for Ethical Decision Makingcited by 55
  9. 2023Generative Social Choicecited by 43
  10. 2014Justified Representation in Approval-Based Committee Votingcited by 42
  11. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  12. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  13. 2020Multi-Winner Voting with Approval Preferencescited by 37
  14. 2020Adapting a Kidney Exchange Algorithm to Align with Human Valuescited by 36
  15. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  16. 2021Survey on reinforcement learning for language processingcited by 31
  17. 2021Hard Choices in Artificial Intelligencecited by 26
  18. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  19. 2023Moral Machine or Tyranny of the Majority?cited by 22
  20. 2015Properties of Multiwinner Voting Rulescited by 20
  21. 2021Choice Set Misspecification in Reward Inferencecited by 13
  22. 2022Representation with Incomplete Votescited by 13
  23. 2024Suppressing Pink Elephants with Direct Principle Feedbackcited by 13
  24. 2020Indecision Modelingcited by 8
  25. 2023Active teacher selection for reward learningcited by 7
  26. 2023RLHF and IIA: Perverse Incentivescited by 6
  27. 2024Cultural Bias in Explainable AI Research: A Systematic Analysiscited by 3
  28. 2019Strategic Voting Under Uncertainty About the Voting Methodcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 32; 2026: 49.

95 papers.

Sort:

By area:

  1. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  2. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  3. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  4. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  5. 2024Self-Pluralising Culture Alignment for Large Language Modelscited by 31
  6. 2024Mapping Social Choice Theory to RLHFcited by 26
  7. 2025AI Alignment at Your Discretioncited by 20
  8. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  9. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  10. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  11. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  12. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matchingcited by 10
38 more
  1. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 9
  2. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHFcited by 9
  3. 2024Trust-Oriented Adaptive Guardrails for Large Language Modelscited by 8
  4. 2024Representative Social Choice: From Learning Theory to AI Alignmentcited by 8
  5. 2024Policy Aggregationcited by 8
  6. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  7. 2025LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spacescited by 8
  8. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  9. 2025Clone-Robust AI Alignmentcited by 7
  10. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Datacited by 7
  11. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  12. 2025Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Valuecited by 6
  13. 2024Learning How to Vote with Principles: Axiomatic Insights Into the Collective Decisions of Neural Networkscited by 5
  14. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  15. 2026Breaking Up with Normatively Monolithic Agency with GRACE: A Reason-Based Neuro-Symbolic Architecture for Safe and Ethical AI Alignmentcited by 5
  16. 2024Abductive and Contrastive Explanations for Scoring Rules in Votingcited by 3
  17. 2024Soft Condorcet Optimization for Ranking of General Agentscited by 3
  18. 2024Pluralistic Alignment Over Timecited by 3
  19. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 3
  20. 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lenscited by 3
  21. 2026Language of Thought Shapes Output Diversity in Large Language Modelscited by 3
  22. 2026Position: Machine Learning for Heart Transplant Allocation Policy Optimization Should Account for Incentivescited by 3
  23. 2025Welfare as a Guiding Principle for Machine Learning -- From Compass, to Lens, to Roadmapcited by 2
  24. 2025Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktalescited by 2
  25. 2025Matching Markets Meet LLMs: Algorithmic Reasoning with Ranked Preferencescited by 2
  26. 2025Reflective Verbal Reward Design for Pluralistic Alignmentcited by 2
  27. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  28. 2025Learning Correlated Reward Models: Statistical Barriers and Opportunitiescited by 2
  29. 2025The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curationcited by 2
  30. 2026Cognitive Models and AI Algorithms Provide Templates for Designing Language Agentscited by 2
  31. 2026Principles Do Not Apply Themselves: A Hermeneutic Perspective on AI Alignmentcited by 2
  32. 2026Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracycited by 2
  33. 2025The Battling Influencers Game: Nash Equilibria Structure of a Potential Game and Implications to Value Alignmentcited by 1
  34. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modelingcited by 1
  35. 2026Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scalecited by 1
  36. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  37. 2026Power and Limitations of Aggregation in Compound AI Systemscited by 1
  38. 2026Finding Common Ground in a Sea of Alternativescited by 1

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
  2. 2025Learning a Canonical Basis of Human Preferences from Binary Ratings
  3. 2024Beyond Preferences in AI Alignment
  4. 2026Align AI to Dynamic Human-AI Workflows
  5. 2024Axioms for AI Alignment from Human Feedback
  6. 2025Adaptive Preference Aggregation
  7. 2024Combining Theory of Mind and Kindness for Self-Supervised Human-AI Alignment
  8. 2026Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values