mlsanityData to 7 Oct 2026

Secrets of RLHF in Large Language Models Part I: PPO

arXiv 2307.04964 ↗First posted 11 Jul 2023cs.CL cs.AI cs.LGcited by 226

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large language models (LLMs) have formulated a blueprint for the advancement of artificial general intelligence. Its primary objective is to function as a human-centric (helpful, honest, and harmless) assistant. Alignment with humans assumes paramount significance, and reinforcement learning with human feedback (RLHF) emerges as the pivotal technological paradigm underpinning this pursuit. Current technical routes usually include reward models to measure human preferences, Proximal Policy Optimization (PPO) to optimize policy model outputs, and process supervision to improve step-by-step reasoning capabilities. However, due to the challenges of reward design, environment interaction, and agent training, coupled with huge trial and error cost of large language models, there is a significant barrier for AI researchers to motivate the development of technical alignment and safe landing of LLMs. The stable training of RLHF has still been a puzzle. In the first report, we dissect the framework of RLHF, re-evaluate the inner workings of PPO, and explore how the parts comprising PPO algorithms impact policy agent training. We identify policy constraints being the key factor for the effective implementation of the PPO algorithm. Therefore, we explore the PPO-max, an advanced version of PPO algorithm, to efficiently improve the training stability of the policy model. Based on our main results, we perform a comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT. The absence of open-source implementations has posed significant challenges to the investigation of LLMs alignment. Therefore, we are eager to release technical reports, reward models and PPO codes, aiming to make modest contributions to the advancement of LLMs.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 46 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

26 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2015Trust Region Policy Optimizationcited by 2,503
11 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023Instruction Tuning with GPT-4cited by 676
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  10. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  11. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 96; 2025: 77; 2026: 36.

226 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  5. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  6. 2023Large Language Model Unlearningcited by 288
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  9. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  10. 2023Challenges and Applications of Large Language Modelscited by 251
  11. 2023Large Language Model Alignment: A Surveycited by 241
  12. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
38 more
  1. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  4. 2023Mitigating the Alignment Tax of RLHFcited by 181
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  6. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  7. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 160
  8. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  9. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  10. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  11. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  12. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  13. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  14. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  15. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  16. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  17. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  18. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  19. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  20. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 82
  21. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  22. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  23. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  24. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  25. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 68
  26. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 68
  27. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  28. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 65
  29. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
  30. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  31. 2024Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Explorationcited by 58
  32. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  33. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 57
  34. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  35. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  36. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  37. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  38. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48

50 most cited of 226 All 226 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  4. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  5. 2023Proximal Policy Optimization Actual Combat: Manipulating Output Tokenizer Length
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  7. 2024Reward-Robust RLHF in LLMs
  8. 2024Accelerated Preference Optimization for Large Language Model Alignment