mlsanityData to 7 Oct 2026

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

arXiv 2404.08555 ↗First posted 12 Apr 2024cs.LG cs.AI cs.CLcited by 89

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

State-of-the-art large language models (LLMs) have become indispensable tools for various tasks. However, training LLMs to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences and mitigate issues like toxicity and hallucinations. Yet, an understanding of RLHF for LLMs is largely entangled with initial design choices that popularized the method and current research focuses on augmenting those choices rather than fundamentally improving the framework. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals, dedicating substantial focus to the core component of RLHF -- the reward model. Our study investigates modeling choices, caveats of function approximation, and their implications on RLHF training algorithms, highlighting the underlying assumptions made about the expressivity of reward. Our analysis improves the understanding of the role of reward models and methods for their training, concurrently revealing limitations of the current methodology. We characterize these limitations, including incorrect generalization, model misspecification, and the sparsity of feedback, along with their impact on the performance of a language model. The discussion and analysis are substantiated by a categorical review of current literature, serving as a reference for researchers and practitioners to understand the challenges of RLHF and build upon existing efforts.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 163 of its 223 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

163 of 223 references.

Closest ancestors · 26

Same direction, newest first

  1. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling14 Feb
  2. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  3. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model14 Oct
  4. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct
  5. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  6. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  7. 2023Efficient RLHF: Reducing the Memory Usage of PPO1 Sep
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
14 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  3. 2023Active Reward Learning from Multiple Teachers2 Mar
  4. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers10 Feb
  5. 2023Chain of Hindsight Aligns Language Models with Feedback6 Feb
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Models of human preference for learning reward functions5 Jun
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  12. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  13. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning27 May
  14. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 137

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  11. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  13. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  15. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  16. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  17. 2017Hindsight Experience Replaycited by 1,096
  18. 2023LIMA: Less Is More for Alignmentcited by 1,080
  19. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  20. 2022Red Teaming Language Models with Language Modelscited by 983
  21. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  22. 2023BloombergGPT: A Large Language Model for Financecited by 923
  23. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  24. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  25. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  26. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  27. 2023Whose Opinions Do Language Models Reflect?cited by 759
  28. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  29. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  30. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  31. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  32. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  33. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  34. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  35. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  36. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  37. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  38. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408

50 most cited of 137 All 137 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 35; 2026: 36.

89 papers.

Sort:

By area:

  1. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 72
  2. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  3. 2024Prompt Optimization with Human Feedbackcited by 35
  4. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  5. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  6. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 28
  7. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 27
  8. 2025Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wildcited by 24
  9. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  10. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  11. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Modelscited by 19
  12. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
38 more
  1. 2024Advanced Natural-based interaction for the ITAlian language: LLaMAntino-3-ANITAcited by 17
  2. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  3. 2025Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Modelscited by 12
  4. 2025CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewardscited by 10
  5. 2024Diversity-Rewarded CFG Distillationcited by 9
  6. 2024Interpreting Language Reward Models via Contrastive Explanationscited by 9
  7. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  8. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Trainingcited by 9
  9. 2025Thinking Preference Optimizationcited by 8
  10. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  11. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  12. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignmentcited by 7
  13. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedbackcited by 7
  14. 2025Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learningcited by 7
  15. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generationcited by 7
  16. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Losscited by 6
  17. 2024MIRROR: A Novel Approach for the Automated Evaluation of Open-Ended Question Generationcited by 6
  18. 2025Length-Controlled Margin-Based Preference Optimization without Reference Modelcited by 6
  19. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual contextcited by 6
  20. 2026Uncertainty-Aware Clarification in LLM Agents with Information Gaincited by 6
  21. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  22. 2025Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHFcited by 5
  23. 2025RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenderscited by 5
  24. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 5
  25. 2024Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challengescited by 4
  26. 2025Active Human Feedback Collection via Neural Contextual Dueling Banditscited by 4
  27. 2025Disaster Management in the Era of Agentic AI Systems: A Vision for Collective Human-Machine Intelligence for Augmented Resiliencecited by 4
  28. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  29. 2026OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Modelscited by 4
  30. 2024Fool Me, Fool Me: User Attitudes Toward LLM Falsehoodscited by 3
  31. 2025Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AIcited by 3
  32. 2026PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMscited by 3
  33. 2024The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcarecited by 2
  34. 2025Aligning Extraction and Generation for Robust Retrieval-Augmented Generationcited by 2
  35. 2025The Structural Safety Generalization Problemcited by 2
  36. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  37. 2025Baba is LLM: Reasoning in a Game with Dynamic Rulescited by 2
  38. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2023The History and Risks of Reinforcement Learning and Human Feedback
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  4. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond
  5. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  6. 2024How to Evaluate Reward Models for RLHF
  7. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  8. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization