mlsanityData to 7 Oct 2026

Fine-tuning language models to find agreement among humans with diverse preferences

arXiv 2211.15006 ↗First posted 28 Nov 2022cs.LG cs.CLcited by 189

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Recent work in large language modeling (LLMs) has used fine-tuning to align outputs with the preferences of a prototypical user. This work assumes that human preferences are static and homogeneous across individuals, so that aligning to a a single "generic" user will confer more general alignment. Here, we embrace the heterogeneity of human preferences to consider a different challenge: how might a machine help people with diverse views find agreement? We fine-tune a 70 billion parameter LLM to generate statements that maximize the expected approval for a group of people with potentially diverse opinions. Human participants provide written opinions on thousands of questions touching on moral and political issues (e.g., "should we raise taxes on the rich?"), and rate the LLM's generated candidate consensus statements for agreement and quality. A reward model is then trained to predict individual preferences, enabling it to quantify and rank consensus statements in terms of their appeal to the overall group, defined according to different aggregation (social welfare) functions. The model produces consensus statements that are preferred by human users over those from prompted LLMs (>70%) and significantly outperforms a tight fine-tuned baseline that lacks the final ranking step. Further, our best model's consensus statements are preferred over the best human-generated opinions (>65%). We find that when we silently constructed consensus statements from only a subset of group members, those who were excluded were more likely to dissent, revealing the sensitivity of the consensus to individual contributions. These results highlight the potential to use LLMs to help groups of humans align their values with one another.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 71 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

31 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
18 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2018Universal Sentence Encodercited by 485
  6. 2020Artificial Intelligence, Values and Alignmentcited by 443
  7. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  8. 2022Teaching language models to support answers with verified quotescited by 290
  9. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  10. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  11. 2015Order Matters: Sequence to sequence for setscited by 236
  12. 2021Alignment of Language Agentscited by 167
  13. 2019Sherlock: A Deep Learning Approach to Semantic Data Type Detectioncited by 73
  14. 2018VizML: A Machine Learning Approach to Visualization Recommendationcited by 38
  15. 2020OpinionDigest: A Simple Framework for Opinion Summarizationcited by 37
  16. 2019Informative and Controllable Opinion Summarizationcited by 27
  17. 2021Modelling Cooperation in Network Games with Spatio-Temporal Complexitycited by 5
  18. 2019DADI: Dynamic Discovery of Fair Information with Adversarial Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 44; 2024: 67; 2025: 42; 2026: 35.

189 papers.

Sort:

By area:

  1. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  2. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  6. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  7. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  8. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  9. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  10. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  11. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  12. 2024A Roadmap to Pluralistic Alignmentcited by 175
38 more
  1. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  2. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  3. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  4. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  5. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  6. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  7. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  8. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  9. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  10. 2024Building Machines that Learn and Think with Peoplecited by 73
  11. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  12. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  13. 2023Identifying and Mitigating Vulnerabilities in LLM-Integrated Applicationscited by 54
  14. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  15. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  16. 2023Learning to Generate Better Than Your LLMcited by 49
  17. 2024Beyond Preferences in AI Alignmentcited by 46
  18. 2023Generative Social Choicecited by 43
  19. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  20. 2023Opportunities and Risks of LLMs for Scalable Deliberation with Poliscited by 41
  21. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  22. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  23. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  24. 2024Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statementscited by 33
  25. 2024Large Language Models for Education: A Surveycited by 32
  26. 2024Participation in the age of foundation modelscited by 32
  27. 2024Calibrating Long-form Generations from Large Language Modelscited by 31
  28. 2023Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literaturecited by 29
  29. 2024A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedianscited by 29
  30. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templatescited by 29
  31. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
  32. 2023Working Memory Capacity of ChatGPT: An Empirical Studycited by 23
  33. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  34. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  35. 2023Align on the Fly: Adapting Chatbot Behavior to Established Normscited by 22
  36. 2024Retrieval Augmented Generation for Domain-specific Question Answeringcited by 22
  37. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  38. 2024ComPO: Community Preferences for Language Model Personalizationcited by 19

50 most cited of 189 All 189 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior
  2. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Law
  3. 2024Dissecting Human and LLM Preferences
  4. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset
  5. 2024Aligning to Thousands of Preferences via System Message Generalization
  6. 2023On Diversified Preferences of Large Language Model Alignment
  7. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
  8. 2024Aligning LLMs with Individual Preferences via Interaction