mlsanityData to 7 Oct 2026

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

arXiv 2403.17031 ↗First posted 24 Mar 2024cs.LGcited by 49

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF) scaling behaviors reported in OpenAI's seminal TL;DR summarization work. We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains in response quality that scale with model size, with our 2.8B, 6.9B models outperforming OpenAI's released 1.3B checkpoint. We publicly release the trained model checkpoints and code to facilitate further research and accelerate progress in the field (urlhttps://github.com/vwxyzjn/summarizefromfeedbackdetails).

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 31 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

17 of 31 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2024Mixtral of Expertscited by 1,785
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  11. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  12. 2023Language Model Alignment with Elastic Resetcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 28; 2026: 6.

49 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  4. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  5. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 95
  6. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  7. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  8. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  9. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  10. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  11. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
  12. 2025ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimizationcited by 50
37 more
  1. 2024Variational Best-of-N Alignmentcited by 49
  2. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  3. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  4. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  5. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
  6. 2025Learning to Reason at the Frontier of Learnabilitycited by 20
  7. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  8. 2024Sample-Efficient Alignment for LLMscited by 16
  9. 2024Learning to Retrieve Iteratively for In-Context Learningcited by 13
  10. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  11. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  12. 2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 13
  13. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  14. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9
  15. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 9
  16. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  17. 2024A Practical Analysis of Human Alignment with *POcited by 7
  18. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimizationcited by 6
  19. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedbackcited by 6
  20. 2025RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Modelscited by 5
  21. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  22. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
  23. 2026A Statistical Framework for Alignment with Biased AI Feedbackcited by 4
  24. 2024Science Out of Its Ivory Tower: Improving Accessibility with Reinforcement Learningcited by 3
  25. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  26. 2025Reranking-based Generation for Unbiased Perspective Summarizationcited by 2
  27. 2025KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHFcited by 2
  28. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  29. 2025PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolationcited by 1
  30. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
  31. 2025CLARIFID: Improving Radiology Report Generation by Reinforcing Clinically Accurate Impressions and Enforcing Detailed Findingscited by 1
  32. 2025RankPO: Preference Optimization for Job-Talent Matchingcited by 0
  33. 2025Failure Modes of Maximum Entropy RLHFcited by 0
  34. 2025Towards better dense rewards in Reinforcement Learning Applicationscited by 0
  35. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
  36. 2026Theoretical Limits of Language Model Alignmentcited by 0
  37. 2026MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  2. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2023A Survey of Reinforcement Learning from Human Feedback
  8. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions