mlsanityData to 7 Oct 2026

Optimizing RLHF Training for Large Language Models with Stage Fusion

arXiv 2409.13221 ↗First posted 20 Sep 2024cs.LG cs.CL cs.DCcited by 41

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We present RLHFuse, an efficient training system with stage fusion for Reinforcement Learning from Human Feedback (RLHF). Due to the intrinsic nature of RLHF training, i.e., the data skewness in the generation stage and the pipeline bubbles in the training stage, existing RLHF systems suffer from low GPU utilization. RLHFuse breaks the traditional view of RLHF workflow as a composition of individual tasks, splitting each task into finer-grained subtasks, and performing stage fusion to improve GPU utilization. RLHFuse contains two key ideas. First, for generation and inference tasks, RLHFuse splits them into sample-level subtasks, enabling efficient inter-stage fusion to overlap the execution of generation and inference stages, thus mitigating the original generation bottleneck dominated by long-tailed samples. Second, for training tasks, RLHFuse breaks them into subtasks of micro-batches and performs intra-stage fusion to concurrently execute these subtasks in the training stage with a fused pipeline schedule, effectively mitigating the pipeline bubbles. The experiments show that RLHFuse increases the training throughput by up to 3.7×, compared to existing systems.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 49 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

35 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation20 Jun
  2. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
20 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  5. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  7. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  8. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  9. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  10. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  11. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  12. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70
  13. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  14. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  15. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  16. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  17. 2023Zero Bubble Pipeline Parallelismcited by 39
  18. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 22
  19. 2022On Optimizing the Communication of Model Parallelismcited by 19
  20. 2024DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 22.

41 papers.

Sort:

By area:

  1. 2026MiMo-V2-Flash Technical Reportcited by 163
  2. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  3. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 48
  4. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  5. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 35
  6. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 27
  7. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  8. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 14
  9. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  10. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 11
  11. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 10
  12. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
29 more
  1. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Trainingcited by 8
  2. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Explorationcited by 7
  3. 2024SELU: Self-Learning Embodied MLLMs in Unknown Environmentscited by 6
  4. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 6
  5. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  6. 2025RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Draftingcited by 4
  7. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Schedulingcited by 4
  8. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  9. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlapcited by 3
  10. 2026SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cachecited by 3
  11. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learningcited by 3
  12. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 2
  13. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learningcited by 2
  14. 2026Faster Synchronous On-Policy RL via Straggler-Aware Group Sizingcited by 2
  15. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 2
  16. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 2
  17. 2026When the Model Said 'No Comment', We Knew Helpfulness Was Dead, Honesty Was Alive, and Safety Was Terrifiedcited by 1
  18. 2026RLHFless: Serverless Computing for Efficient RLHFcited by 1
  19. 2026AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMscited by 1
  20. 2026Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelismcited by 1
  21. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switchcited by 1
  22. 2025Alchemist: Towards the Design of Efficient Online Continual Learning Systemcited by 0
  23. 2025Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novelscited by 0
  24. 2026ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learningcited by 0
  25. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 0
  26. 2026Conduit: An Experience Data Plane for Distributed Reinforcement Learningcited by 0
  27. 2026WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learningcited by 0
  28. 2026Nereus: Adaptive Parallelism for LLM Post-Trainingcited by 0
  29. 2026RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
  2. 2024The Perfect Blend: Redefining RLHF with Mixture of Judges
  3. 2024HybridFlow: A Flexible and Efficient RLHF Framework
  4. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  5. 2025G-Core: A Simple, Scalable and Balanced RLHF Trainer
  6. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
  7. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHF