mlsanityData to 14 Oct 2026

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

arXiv 2405.11143 ↗First posted 20 May 2024cs.AI cs.CL cs.LGcited by 73

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) significantly improve the alignment of human-AI values, further raising the upper bound of AI capabilities, particularly in reasoning-intensive, long-context Chain-of-Thought (CoT) tasks. However, existing frameworks commonly face challenges such as inference bottlenecks and complexity barriers, which restrict their accessibility to newcomers. To bridge this gap, we introduce OpenRLHF, a user-friendly, scalable, and easy-to-learn open-source RLHF framework built upon Ray, vLLM, DeepSpeed, and HuggingFace Transformers, featuring a simplified design, clear code structure, and comprehensive documentation to facilitate entry for researchers and practitioners. Experimental results show that OpenRLHF achieves superior training efficiency, with speedups ranging from 1.22x to 1.68x across different model sizes, compared to state-of-the-art frameworks. Additionally, it requires significantly fewer lines of code for implementation. OpenRLHF is publicly available at https://github.com/OpenRLHF/OpenRLHF, and has already been adopted by leading institutions to accelerate RLHF research and learning.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 31 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

18 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Let's Verify Step by Step31 May
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  12. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
3 more
  1. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  2. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  3. 2020Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 47; 2026: 22.

73 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  5. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  6. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  7. 2025Reinforcement Learning from Human Feedbackcited by 58
  8. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  9. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  10. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  11. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  12. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 37
38 more
  1. 2025Learning to Reason for Long-Form Story Generationcited by 34
  2. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  3. 2025ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Datasetcited by 30
  4. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 25
  5. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
  6. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Methodcited by 23
  7. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  8. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 20
  9. 2025R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learningcited by 20
  10. 2025Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineeringcited by 17
  11. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  12. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  13. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasonerscited by 16
  14. 2025Play to Generalize: Learning to Reason Through Game Playcited by 16
  15. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  16. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 15
  17. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  18. 2025Reasoning with Reinforced Functional Token Tuningcited by 14
  19. 2025AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learningcited by 14
  20. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  21. 2025Truth in the Few: High-Value Data Selection for Efficient Multi-Modal Reasoningcited by 11
  22. 2025Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoningcited by 9
  23. 2025Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulationcited by 8
  24. 2025CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learningcited by 6
  25. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  26. 2025From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMscited by 5
  27. 2025Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoningcited by 5
  28. 2025Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solvingcited by 4
  29. 2025A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directionscited by 4
  30. 2025Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Modelcited by 4
  31. 2026Milestones over Outcome: Unlocking Geometric Reasoning with Sub-Goal Verifiable Rewardcited by 4
  32. 2024SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMscited by 3
  33. 2025PathFound: An Agentic Multimodal Model Activating Evidence-seeking Pathological Diagnosiscited by 3
  34. 2026Learning to Reason with Curriculum II: Compositional Generalizationcited by 3
  35. 2025MARGE: Improving Math Reasoning for LLMs with Guided Explorationcited by 2
  36. 2025DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answeringcited by 2
  37. 2025In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedbackcited by 2
  38. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Abilitycited by 2

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  4. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  5. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  7. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
  8. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback