mlsanityData to 14 Oct 2026

Efficient RLHF: Reducing the Memory Usage of PPO

arXiv 2309.00754 ↗First posted 1 Sep 2023cs.LG cs.AI cs.CLcited by 40

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) has revolutionized language modeling by aligning models with human preferences. However, the RL stage, Proximal Policy Optimization (PPO), requires over 3x the memory of Supervised Fine-Tuning (SFT), making it infeasible to use for most practitioners. To address this issue, we present a comprehensive analysis the memory usage, performance, and training time of memory-savings techniques for PPO. We introduce Hydra-RLHF by first integrating the SFT and Reward models and then dynamically turning LoRA "off" during training. Our experiments show: 1. Using LoRA during PPO reduces its memory usage to be smaller than SFT while improving alignment across four public benchmarks, and 2. Hydra-PPO reduces the latency per sample of LoRA-PPO by up to 65% while maintaining its performance. Our results demonstrate that Hydra-PPO is a simple and promising solution for enabling more widespread usage of RLHF.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 51 references that are in the map. First posted on arXiv 1 Sep 2023.

Builds on

39 of 51 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  8. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
18 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  5. 2023Textbooks Are All You Needcited by 569
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  7. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  8. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  9. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  11. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  12. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  13. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  14. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  15. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  16. 2023Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMscited by 38
  17. 2023A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Modelcited by 19
  18. 2019MERL: Multi-Head Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 17; 2025: 13; 2026: 5.

40 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  3. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  6. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 172
  7. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  8. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  9. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  10. 2024A Survey on LoRA of Large Language Modelscited by 67
  11. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 63
  12. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
28 more
  1. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  2. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 42
  3. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  4. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  5. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 21
  6. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFcited by 20
  7. 2023Adapting LLM Agents with Universal Feedback in Communicationcited by 19
  8. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  9. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  10. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 15
  11. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
  12. 2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competenciescited by 9
  13. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Contentcited by 9
  14. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  15. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimizationcited by 7
  16. 2024Atoxia: Red-teaming Large Language Models with Target Toxic Answerscited by 7
  17. 2025Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Modelscited by 7
  18. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimizationcited by 6
  19. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimizationcited by 4
  20. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  21. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithmscited by 1
  22. 2026PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answeringcited by 1
  23. 2026Latent Adversarial Regularization for Offline Preference Optimizationcited by 1
  24. 2026Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descentcited by 1
  25. 2024XXAI: Towards eXplicitly eXplainable Artificial Intelligencecited by 0
  26. 2025Balancing Global Quality and Pronoun-Specific Feedback for Context-Aware Machine Translationcited by 0
  27. 2026Not How Many, But Which: Parameter Placement in Low-Rank Adaptationcited by 0
  28. 2026Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Parameter Efficient Reinforcement Learning from Human Feedback
  2. 2024Understanding and Alleviating Memory Consumption in RLHF for LLMs
  3. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  4. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
  5. 2026PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF
  6. 2023Proximal Policy Optimization Actual Combat: Manipulating Output Tokenizer Length
  7. 2026Memory-Efficient Policy Libraries with Low-Rank Adaptation in Reinforcement Learning
  8. 2023Secrets of RLHF in Large Language Models Part I: PPO