mlsanityData to 7 Oct 2026

HybridFlow: A Flexible and Efficient RLHF Framework

arXiv 2409.19256 ↗First posted 28 Sep 2024cs.LG cs.DCcited by 1,980

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is widely used in Large Language Model (LLM) alignment. Traditional RL can be modeled as a dataflow, where each node represents computation of a neural network (NN) and each edge denotes data dependencies between the NNs. RLHF complicates the dataflow by expanding each node into a distributed LLM training or generation program, and each edge into a many-to-many multicast. Traditional RL frameworks execute the dataflow using a single controller to instruct both intra-node computation and inter-node communication, which can be inefficient in RLHF due to large control dispatch overhead for distributed intra-node computation. Existing RLHF systems adopt a multi-controller paradigm, which can be inflexible due to nesting distributed computation and data communication. We propose HybridFlow, which combines single-controller and multi-controller paradigms in a hybrid manner to enable flexible representation and efficient execution of the RLHF dataflow. We carefully design a set of hierarchical APIs that decouple and encapsulate computation and data dependencies in the complex RLHF dataflow, allowing efficient operation orchestration to implement RLHF algorithms and flexible mapping of the computation onto various devices. We further design a 3D-HybridEngine for efficient actor model resharding between training and generation phases, with zero memory redundancy and significantly reduced communication overhead. Our experimental results demonstrate 1.53×~20.57× throughput improvement when running various RLHF algorithms using HybridFlow, as compared with state-of-the-art baselines. HybridFlow source code will be available at https://github.com/volcengine/verl.

Key facts. Cited by 1,980 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 89 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

47 of 89 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization24 Mar
  2. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  3. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learning18 Oct
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  5. 2023Efficient RLHF: Reducing the Memory Usage of PPO1 Sep
  6. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
26 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  7. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  9. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  10. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  11. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  13. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  14. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  15. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  16. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  17. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  18. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  19. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  20. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 40
  21. 2021A Runtime-Based Computational Performance Predictor for Deep Neural Network Trainingcited by 29
  22. 2020RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problemcited by 15
  23. 2017TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlowcited by 9
  24. 2024HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesiscited by 8
  25. 2023SWIFT: Expedited Failure Recovery for Large-scale DNN Trainingcited by 5
  26. 2023GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 795; 2026: 1180.

1,980 papers.

Sort:

By area:

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  7. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  9. 2025Process Reinforcement through Implicit Rewardscited by 478
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  11. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  12. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
38 more
  1. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  4. 2026Reinforcement Learning via Self-Distillationcited by 285
  5. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  6. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  7. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  8. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  9. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  10. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  11. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  12. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
  13. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  14. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  15. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  16. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 182
  17. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  18. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  19. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  20. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  21. 2025Agentic Reinforced Policy Optimizationcited by 166
  22. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  23. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  24. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  25. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  26. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  27. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  28. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  29. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  30. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  31. 2025Phi-4-reasoning Technical Reportcited by 146
  32. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  33. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  34. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 141
  35. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  36. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  37. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  38. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130

50 most cited of 1,980 All 1,980 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
  2. 2026RLHFless: Serverless Computing for Efficient RLHF
  3. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
  4. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  5. 2024Optimizing RLHF Training for Large Language Models with Stage Fusion
  6. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  8. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions