mlsanityData to 7 Oct 2026

Reinforcement Learning from Human Feedback

arXiv 2504.12501 ↗First posted 16 Apr 2025cs.LGcited by 53

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has become a crucial tool to build the latest machine learning systems at scale. The field grew around the core methods of RLHF into today's broader suite of post-training techniques. In this book, we give a comprehensive introduction to the core methods for post-training models for people with some level of quantitative background, organized around the canonical RLHF recipe. The book starts with what RLHF does and why it was created, with seminal technical milestones in its young history and a primer on reinforcement learning context needed to understand the book. The core of the book details every optimization stage in using RLHF, from starting with instruction tuning to training a reward model and finally all of rejection sampling, reinforcement learning, on-policy distillation, and direct alignment algorithms. The book also discusses broader topics, such as the origins of RLHF -- both in recent literature and in a convergence of disparate fields of science in economics, philosophy, and optimal control. The book concludes with advanced topics -- understudied or emerging research questions in synthetic data, tool-use, character training, and evaluation -- and open questions for the field. The book is released with a variety of companion resources, including a codebase, a library to compare model completions from within post-training stages, and an educational course, to be a one-stop shop for learning all foundational concepts for post-training language models.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 110 of its 417 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

110 of 417 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  2. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs14 Mar
  3. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent24 Feb
  4. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysis18 Feb
  5. 2024Critique-out-Loud Reward Models21 Aug
  6. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  7. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  9. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models22 Jan
  10. 2024Self-Rewarding Language Models18 Jan
  11. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
6 more
  1. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2018Scalable agent alignment via reward modeling: a research direction19 Nov

Other references · 92

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
38 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2025s1: Simple test-time scalingcited by 1,311
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  13. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  14. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  15. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  16. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  17. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  18. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  19. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  20. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  21. 2024DeepSeek-V3 Technical Reportcited by 724
  22. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  23. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  24. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  25. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  26. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  27. 2025Humanity's Last Examcited by 467
  28. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  29. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  30. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  31. 20242 OLMo 2 Furiouscited by 302
  32. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  33. 2022Teaching language models to support answers with verified quotescited by 290
  34. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  35. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
  36. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  37. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  38. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 32.

53 papers.

Sort:

By area:

  1. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  3. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  4. 2025Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIcited by 19
  5. 2025Guiding Generative Models to Uncover Diverse and Novel Crystals via Reinforcement Learningcited by 19
  6. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  7. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  8. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  9. 2025Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Supportcited by 10
  10. 2025Using Reinforcement Learning to Train Large Language Models to Explain Human Decisionscited by 8
  11. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  12. 2025Survey: Multi-Armed Bandits Meet Large Language Modelscited by 6
38 more
  1. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuningcited by 6
  2. 2026Generative AI collective behavior needs an interactionist paradigmcited by 6
  3. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  4. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  5. 2026SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Carecited by 5
  6. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  7. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  8. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradientscited by 4
  9. 2026Operationalising the Superficial Alignment Hypothesis via Task Complexitycited by 4
  10. 2026Personalization Increases Affective Alignment but Has Role-Dependent Effects on Epistemic Independence in LLMscited by 4
  11. 2026Steering LLMs for Culturally Localized Generationcited by 4
  12. 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
  13. 2025DSCD: Large Language Model Detoxification with Self-Constrained Decodingcited by 3
  14. 2025Sophia: A Persistent Agent Framework of Artificial Lifecited by 3
  15. 2026Object Hallucination-Free Reinforcement Unlearning for Vision-Language Modelscited by 3
  16. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressurecited by 3
  17. 2025Advances in LLMs with Focus on Reasoning, Adaptability, Efficiency and Ethicscited by 2
  18. 2025Physics of Learning: A Lagrangian perspective to different learning paradigmscited by 2
  19. 2026Indic-TunedLens: Interpreting Multilingual Models in Indian Languagescited by 2
  20. 2026Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Userscited by 2
  21. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verificationcited by 2
  22. 2025TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleavingcited by 1
  23. 2026Reinforced Curriculum Pre-Alignment for Domain-Adaptive VLMscited by 1
  24. 2026Adversarial Network Imagination: Causal LLMs and Digital Twins for Proactive Telecom Mitigationcited by 1
  25. 2026In Transformer We Trust? A Perspective on Transformer Architecture Failure Modescited by 1
  26. 2026Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontiercited by 1
  27. 2026Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationscited by 1
  28. 2025SmolKalam: Ensemble Quality-Filtered Translation at Scale for High Quality Arabic Post-Training Datacited by 0
  29. 2026Jailbreaking LLMs Without Gradients or Priors: Effective and Transferable Attackscited by 0
  30. 2026Latency-aware Human-in-the-Loop Reinforcement Learning for Semantic Communicationscited by 0
  31. 2026DEEP: Docker-based Execution and Evaluation Platformcited by 0
  32. 2026Fuel Gauge: Estimating Chain-of-Thought Length Ahead of Time in Large Multimodal Modelscited by 0
  33. 2026Beyond Distribution Sharpening: The Importance of Task Rewardscited by 0
  34. 2026MoRFI: Monotonic Sparse Autoencoder Feature Identificationcited by 0
  35. 2026Truthful Online Preference Aggregation for LLM Fine-Tuning in Mobile Crowdsourcingcited by 0
  36. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0
  37. 2026PS-PPO: Prefix-Sampling PPO for Critic-Free RLHFcited by 0
  38. 2026From Detection to Refusal: Safer LLMs via Circuit-Guided Weight Scalingcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey of Reinforcement Learning from Human Feedback
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  5. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  8. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences