mlsanityData to 14 Oct 2026

Understanding and Alleviating Memory Consumption in RLHF for LLMs

arXiv 2410.15651 ↗First posted 21 Oct 2024cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Fine-tuning with Reinforcement Learning with Human Feedback (RLHF) is essential for aligning large language models (LLMs). However, RLHF often encounters significant memory challenges. This study is the first to examine memory usage in the RLHF context, exploring various memory management strategies and unveiling the reasons behind excessive memory consumption. Additionally, we introduce a simple yet effective approach that substantially reduces the memory required for RLHF fine-tuning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 21 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

14 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training19 Dec
  2. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug

Other references · 12

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  9. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  10. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  11. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  12. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient RLHF: Reducing the Memory Usage of PPO
  2. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  4. 2024Measuring memorization in RLHF for code completion
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity
  6. 2024Reinforcement Learning Enhanced LLMs: A Survey
  7. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
  8. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity