mlsanityData to 7 Oct 2026

A Survey of Reinforcement Learning from Human Feedback

arXiv 2312.14925 ↗First posted 22 Dec 2023cs.LGcited by 259

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that learns from human feedback instead of relying on an engineered reward function. Building on prior work on the related setting of preference-based reinforcement learning (PbRL), it stands at the intersection of artificial intelligence and human-computer interaction. This positioning provides a promising approach to enhance the performance and adaptability of intelligent systems while also improving the alignment of their objectives with human values. The success in training large language models (LLMs) has impressively demonstrated this potential in recent years, where RLHF has played a decisive role in directing the model's capabilities towards human objectives. This article provides an overview of the fundamentals of RLHF, exploring how RL agents interact with human feedback. While recent focus has been on RLHF for LLMs, our survey covers the technique across multiple domains. We provide our most comprehensive coverage in control and robotics, where many fundamental techniques originate, alongside a dedicated LLM section. We examine the core principles that underpin RLHF, how algorithms and human feedback work together, and the main research trends in the field. Our goal is to give researchers and practitioners a clear understanding of this rapidly growing field.

Key facts. Cited by 259 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 424 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

45 of 424 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  2. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  3. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems24 Jul
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  5. 2022Time-Efficient Reward Learning via Visually Assisted Cluster Ranking30 Nov
  6. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback21 Nov
  7. 2022Reward Learning with Trees: Methods and Evaluation3 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2022Uncertainty Estimation for Language Reward Models14 Mar
  10. 2022Safe Deep RL in 3D Environments using Human Feedback20 Jan
  11. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  12. 2021Preference-based Learning of Reward Function Features3 Mar
3 more
  1. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  2. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  3. 2018Scalable agent alignment via reward modeling: a research direction19 Nov

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  11. 2022Self-critiquing models for assisting human evaluatorscited by 372
  12. 2022Teaching language models to support answers with verified quotescited by 290
18 more
  1. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  2. 2023AI Alignment: A Comprehensive Surveycited by 256
  3. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  4. 2018Supervising strong learners by amplifying weak expertscited by 139
  5. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  6. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  7. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 35
  8. 2022imitation: Clean Imitation Learning Implementationscited by 34
  9. 2021Human irrationality: both bad and good for reward inferencecited by 25
  10. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  11. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  12. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
  13. 2021Combining Reward Information from Multiple Sourcescited by 7
  14. 2022Researching Alignment Research: Unsupervised Analysiscited by 7
  15. 2023Active teacher selection for reward learningcited by 7
  16. 2022Methodological reflections for AI alignment research using human feedbackcited by 5
  17. 2022POLAR: Preference Optimization and Learning Algorithms for Roboticscited by 3
  18. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 79; 2025: 129; 2026: 51.

259 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  3. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  4. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  6. 2025Agentic Reinforced Policy Optimizationcited by 166
  7. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  8. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  9. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  10. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  11. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  12. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
38 more
  1. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  3. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  4. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  5. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  6. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  7. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  8. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 63
  9. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 53
  12. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  13. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  14. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 43
  15. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  16. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 41
  17. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  18. 2025TxGemma: Efficient and Agentic LLMs for Therapeuticscited by 40
  19. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
  20. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  21. 2025A Survey of Direct Preference Optimizationcited by 35
  22. 2025Time Series Forecasting via Reasoning: A Slow-Thinking Approach with Reinforcement Fine-Tuned LLMscited by 34
  23. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  24. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 33
  25. 2024ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systemscited by 30
  26. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  27. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  28. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  29. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  30. 2025ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineeringcited by 26
  31. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  32. 2024VideoAgent: Self-Improving Video Generationcited by 24
  33. 2024Causality for Large Language Modelscited by 24
  34. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
  35. 2025ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learningcited by 24
  36. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  37. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  38. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMscited by 23

50 most cited of 259 All 259 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  2. 2025Reinforcement Learning from Human Feedback
  3. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  8. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning