mlsanityData to 14 Oct 2026

ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation

arXiv 2406.14088 ↗First posted 20 Jun 2024cs.DC cs.AI cs.CLcited by 37

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for empowering large language model (LLM) applications. Compared with the supervised training process of LLMs, the RLHF training process is much more sophisticated, requiring a diverse range of computation workloads with intricate dependencies between multiple LLM instances. Therefore, simply adopting the fixed parallelization strategies from supervised training for LLMs can be insufficient for RLHF and result in low training efficiency. To overcome this limitation, we propose a novel technique named parameter ReaLlocation, which dynamically adapts the parallelization strategies for different workloads during training by redistributing LLM parameters across the training cluster. Building upon this idea, we introduce ReaL, a pioneering system for efficient RLHF training. ReaL introduces the concept of an execution plan, which defines a fine-grained resource allocation and parallelization strategy particularly designed for RLHF training. Based on this concept, ReaL employs a tailored search algorithm with a lightweight run-time estimator to automatically discover an efficient execution plan for an instance of RLHF experiment. Subsequently, the runtime engine deploys the selected plan by effectively parallelizing computations and redistributing parameters. We evaluate ReaL on the LLaMA models with up to 70 billion parameters and 128 GPUs. The experimental results demonstrate that ReaL achieves speedups of up to 3.58× compared to baseline methods. Furthermore, the execution plans generated by ReaL exhibit an average of 81% performance improvement over heuristic approaches based on Megatron-LM in the long-context scenario. The source code of ReaL is publicly available at https://github.com/openpsi-project/ReaLHF .

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 59 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

39 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training19 Dec
  2. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  3. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
20 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  8. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  9. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  10. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  11. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  12. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  13. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  14. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  15. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  16. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  17. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  18. 2022On Optimizing the Communication of Model Parallelismcited by 19
  19. 2023SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Corescited by 10
  20. 2024ATOM: Asynchronous Training of Massive Models for Deep Learning in a Decentralized Environmentcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 21.

37 papers.

Sort:

By area:

  1. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 104
  2. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  3. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  4. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 62
  5. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 54
  6. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  7. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  8. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  9. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  10. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifiercited by 15
  11. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 14
  12. 2025OckBench: Measuring the Efficiency of LLM Reasoningcited by 11
25 more
  1. 2025MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Clustercited by 9
  2. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 9
  3. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Schedulingcited by 6
  4. 20247B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancementcited by 5
  5. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  6. 2025A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directionscited by 4
  7. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  8. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  9. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  10. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2
  11. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  12. 2026DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Trainingcited by 2
  13. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rolloutscited by 2
  14. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 1
  15. 2026AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMscited by 1
  16. 2026Nereus: Adaptive Parallelism for LLM Post-Trainingcited by 1
  17. 2025Balancing Robustness and Efficiency in Embedded DNNs Through Activation Function Selectioncited by 0
  18. 2026Physio-DPO: Aligning Large Language Models with the Protein Energy Landscape to Eliminate Structural Hallucinationscited by 0
  19. 2026Graph-Based Chain-of-Thought Pruning for Reducing Redundant Reflections in Reasoning LLMscited by 0
  20. 2026Decomposer: Learning to Decompile Symbolic Music to Programscited by 0
  21. 2026MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Modelscited by 0
  22. 2026QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agentscited by 0
  23. 2026ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learningcited by 0
  24. 2026Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learningcited by 0
  25. 2026DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Losscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  2. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  3. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
  4. 2026RLHFless: Serverless Computing for Efficient RLHF
  5. 2024HybridFlow: A Flexible and Efficient RLHF Framework
  6. 2023Secrets of RLHF in Large Language Models Part I: PPO
  7. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
  8. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework