mlsanityData to 14 Oct 2026

The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values

arXiv 2310.07629 ↗First posted 11 Oct 2023cs.CL cs.CYcited by 54

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Human feedback is increasingly used to steer the behaviours of Large Language Models (LLMs). However, it is unclear how to collect and incorporate feedback in a way that is efficient, effective and unbiased, especially for highly subjective human preferences and values. In this paper, we survey existing approaches for learning from human feedback, drawing on 95 papers primarily from the ACL and arXiv repositories.First, we summarise the past, pre-LLM trends for integrating human feedback into language models. Second, we give an overview of present techniques and practices, as well as the motivations for using feedback; conceptual frameworks for defining values and preferences; and how feedback is collected and from whom. Finally, we encourage a better future of feedback learning in LLMs by raising five unresolved conceptual and practical challenges.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 95 of its 152 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

95 of 152 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Human Feedback is not Gold Standard28 Sep
  2. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar
  3. 2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits1 Jan
  4. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
  5. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning14 Oct
  6. 2022Enabling Classifiers to Make Judgements Explicitly Aligned with Human Values14 Oct
  7. 2022Towards Boosting the Open-Domain Chatbot with Human Feedback30 Aug
  8. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback5 Aug
  9. 2022Training Language Models with Language Feedback29 Apr
  10. 2022Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback12 Apr
  11. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Data15 Sep
  12. 2020Learning to summarize from human feedback2 Sep
3 more
  1. 2020Aligning AI With Shared Human Values5 Aug
  2. 2019Learning Norms from Stories: A Prior for Value Aligned Agents7 Dec
  3. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback3 May

Other references · 80

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
38 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,100
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  4. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  8. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  10. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  11. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  12. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  13. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  14. 2022Teaching language models to support answers with verified quotescited by 292
  15. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  16. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  17. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  18. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  19. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  20. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  21. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  22. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  23. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  24. 2021Challenges in Detoxifying Language Modelscited by 169
  25. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  26. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  27. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  28. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  29. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  30. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  31. 2022Power to the People? Opportunities and Challenges for Participatory AIcited by 138
  32. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  33. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  34. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  35. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 101
  36. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  37. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 93
  38. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 23; 2026: 9.

54 papers.

Sort:

By area:

  1. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  2. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  3. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  4. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  5. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  6. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  7. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  8. 2025A Survey of Direct Preference Optimizationcited by 36
  9. 2024Participation in the age of foundation modelscited by 32
  10. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  11. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 30
  12. 2024A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedianscited by 29
38 more
  1. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 26
  2. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  3. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  4. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 20
  5. 2024ComPO: Community Preferences for Language Model Personalizationcited by 19
  6. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  7. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 14
  8. 2024LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Surveycited by 12
  9. 2024AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligencecited by 12
  10. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 11
  11. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  12. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  13. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  14. 2024How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studiescited by 10
  15. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  16. 2025Learning to summarize user information for personalized reinforcement learning from human feedbackcited by 10
  17. 2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competenciescited by 9
  18. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  19. 2024Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedbackcited by 6
  20. 2024SPICA: Retrieving Scenarios for Pluralistic In-Context Alignmentcited by 6
  21. 2025SPRI: Aligning Large Language Models with Context-Situated Principlescited by 5
  22. 2025AdParaphrase: Paraphrase Dataset for Analyzing Linguistic Features toward Generating Attractive Ad Textscited by 5
  23. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  24. 2025Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signalscited by 5
  25. 2025Political Ideology Shifts in Large Language Modelscited by 5
  26. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  27. 2024Fool Me, Fool Me: User Attitudes Toward LLM Falsehoodscited by 3
  28. 2026Text as a Universal Interface for Transferable Personalizationcited by 3
  29. 2025Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Modelscited by 2
  30. 2026AI Alignment From Social Choice Perspectivescited by 2
  31. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  32. 2024Intuitions of Compromise: Utilitarianism vs. Contractualismcited by 1
  33. 2025Ideology-Based LLMs for Content Moderationcited by 1
  34. 2026CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Taskscited by 1
  35. 2026Growth First, Care Second? Tracing the Landscape of LLM Value Preferences in Everyday Dilemmascited by 1
  36. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  37. 2025Bridging Subjective and Objective QoE: Operator-Level Aggregation Using LLM-Based Comment Analysis and Network MOS Comparisoncited by 0
  38. 2025Survey-to-Behavior: Downstream Alignment of Human Values in LLMs via Survey Questionscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Let Me Teach You: Pedagogical Foundations of Feedback for Language Models
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  4. 2024Aligning language models with human preferences
  5. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
  6. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  7. 2024Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions
  8. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation