mlsanityData to 7 Oct 2026

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

arXiv 2307.15217 ↗First posted 27 Jul 2023cs.AI cs.CL cs.LGcited by 743

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of a multi-faceted approach to the development of safer AI systems.

Key facts. Cited by 743 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 192 of its 292 references that are in the map. First posted on arXiv 27 Jul 2023.

Builds on

192 of 292 references.

Closest ancestors · 28

Same direction, newest first

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023Reward Collapse in Aligning Large Language Models28 May
  5. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models19 May
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  8. 2023Active Reward Learning from Multiple Teachers2 Mar
  9. 2023Internally Rewarded Reinforcement Learning1 Feb
  10. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  11. 2023On The Fragility of Learned Reward Functions9 Jan
  12. 2022The Expertise Problem: Learning from Specialized Feedback12 Nov
16 more
  1. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  2. 2022Models of human preference for learning reward functions5 Jun
  3. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  4. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2022Uncertainty Estimation for Language Reward Models14 Mar
  7. 2021Learning Multimodal Rewards from Rankings27 Sep
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  9. 2021Choice Set Misspecification in Reward Inference19 Jan
  10. 2020Learning Rewards from Linguistic Feedback30 Sep
  11. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  13. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  14. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  15. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  16. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 164

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2023LIMA: Less Is More for Alignmentcited by 1,080
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2022Mass-Editing Memory in a Transformercited by 976
  9. 2021Ethical and social risks of harm from Language Modelscited by 957
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  13. 2023Large Language Modelscited by 774
  14. 2023Whose Opinions Do Language Models Reflect?cited by 759
  15. 2017Deep Reinforcement Learning that Matterscited by 727
  16. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  17. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  18. 2018AI safety via debatecited by 384
  19. 2022Self-critiquing models for assisting human evaluatorscited by 372
  20. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  21. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  22. 2023How Language Model Hallucinations Can Snowballcited by 310
  23. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  24. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  25. 2023Are aligned neural networks adversarially aligned?cited by 305
  26. 2016Cooperative Inverse Reinforcement Learningcited by 298
  27. 2021Unsolved Problems in ML Safetycited by 291
  28. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  29. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  30. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  31. 2023Poisoning Language Models During Instruction Tuningcited by 254
  32. 2017Inverse Reward Designcited by 246
  33. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  34. 2023Pretraining Language Models with Human Preferencescited by 245
  35. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  36. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  37. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  38. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209

50 most cited of 164 All 164 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 68; 2024: 266; 2025: 238; 2026: 171.

743 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  3. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  5. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  8. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  9. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  11. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  12. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
38 more
  1. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  2. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  3. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  4. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  5. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  8. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  9. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  10. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  11. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  12. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  13. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  14. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  15. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  16. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  17. 2023Mitigating the Alignment Tax of RLHFcited by 181
  18. 2024A Roadmap to Pluralistic Alignmentcited by 175
  19. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  20. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  21. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  22. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  23. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  24. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  25. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  26. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  27. 2023Getting aligned on representational alignmentcited by 142
  28. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  29. 2024LLM Critics Help Catch LLM Bugscited by 138
  30. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  31. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  32. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  33. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  34. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  35. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
  36. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  37. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  38. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121

50 most cited of 743 All 743 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey of Reinforcement Learning from Human Feedback
  2. 2025Reinforcement Learning from Human Feedback
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  4. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond
  5. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  6. 2023Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback