mlsanityData to 7 Oct 2026

The History and Risks of Reinforcement Learning and Human Feedback

arXiv 2310.13595 ↗First posted 20 Oct 2023cs.CYcited by 42

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to use and more effective. A core piece of the RLHF process is the training and utilization of a model of human preferences that acts as a reward function for optimization. This approach, which operates at the intersection of many stakeholders and academic disciplines, remains poorly understood. RLHF reward models are often cited as being central to achieving performance, yet very few descriptors of capabilities, evaluations, training methods, or open-source models exist. Given this lack of information, further study and transparency is needed for learned RLHF reward models. In this paper, we illustrate the complex history of optimizing preferences, and articulate lines of inquiry to understand the sociotechnical context of reward models. In particular, we highlight the ontological differences between costs, rewards, and preferences at stake in RLHF's foundations, related methodological tensions, and possible research directions to improve general understanding of how reward models function.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 86 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

46 of 86 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF28 Sep
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  10. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  12. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
23 more
  1. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  2. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  3. 2016Cooperative Inverse Reinforcement Learningcited by 298
  4. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  5. 2022Teaching language models to support answers with verified quotescited by 290
  6. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  7. 2023The False Promise of Imitating Proprietary LLMscited by 254
  8. 2023Self-Consuming Generative Models Go MADcited by 239
  9. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  10. 2023Shepherd: A Critic for Language Model Generationcited by 96
  11. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  12. 2021On the Expressivity of Markov Rewardcited by 71
  13. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  14. 2019Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approachcited by 32
  15. 2021Hard Choices in Artificial Intelligencecited by 26
  16. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  17. 2023Moral Machine or Tyranny of the Majority?cited by 22
  18. 2017Should Robots be Obedient?cited by 20
  19. 2022Reward Reports for Reinforcement Learningcited by 18
  20. 2022Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systemscited by 12
  21. 2020Multi-Principal Assistance Gamescited by 10
  22. 2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
  23. 2023The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practicescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 26; 2025: 9; 2026: 4.

42 papers.

Sort:

By area:

  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  2. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  5. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  6. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  7. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  8. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  9. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  10. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  11. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  12. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
30 more
  1. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  2. 2025Reinforcement Learning from Human Feedbackcited by 53
  3. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  4. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  5. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
  6. 2024Mapping Social Choice Theory to RLHFcited by 26
  7. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  8. 2024Prototypical Reward Network for Data-Efficient RLHFcited by 22
  9. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  10. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 18
  11. 2024Suppressing Pink Elephants with Direct Principle Feedbackcited by 13
  12. 2024Acceptable Use Policies for Foundation Modelscited by 12
  13. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  14. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  15. 2024Responsible Reporting for Frontier AI Developmentcited by 10
  16. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 10
  17. 2024ConTrans: Weak-to-Strong Alignment Engineering via Concept Transplantationcited by 8
  18. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  19. 2024Towards Safe and Honest AI Agents with Neural Self-Other Overlapcited by 6
  20. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Modelscited by 5
  21. 2025A Review of Incorporating Psychological Theories in LLMscited by 5
  22. 2024Value Alignment from Unstructured Textcited by 4
  23. 2026IR3: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hackingcited by 4
  24. 2025Improved Bounds for Private and Robust Alignmentcited by 3
  25. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  26. 2024Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understandingcited by 2
  27. 2024Intuitions of Compromise: Utilitarianism vs. Contractualismcited by 1
  28. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?cited by 1
  29. 2026AI Alignment From Social Choice Perspectivescited by 0
  30. 2026Long-term Measurements: Towards a Longitudinal Understanding of Human-AI Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  5. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  6. 2024How to Evaluate Reward Models for RLHF
  7. 2025Influencing Humans to Conform to Preference Models for RLHF
  8. 2024Optimal Design for Reward Modeling in RLHF