mlsanityData to 7 Oct 2026

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

arXiv 2503.14476 ↗First posted 18 Mar 2025cs.LG cs.CLcited by 2,678

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.

Key facts. Cited by 2,678 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 37 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

18 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  2. 2025Process Reinforcement through Implicit Rewards3 Feb
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  11. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  12. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
2 more
  1. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  2. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1033; 2026: 1645.

2,678 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  3. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  4. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  7. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  8. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  9. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  10. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  11. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  12. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
38 more
  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  2. 2026Reinforcement Learning via Self-Distillationcited by 285
  3. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  5. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  6. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  8. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  9. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  10. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  11. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  12. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  13. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  14. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  15. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  16. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  17. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  18. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  19. 2025Agentic Reinforced Policy Optimizationcited by 166
  20. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  21. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  22. 2026On-Policy Context Distillation for Language Modelscited by 161
  23. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  24. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  25. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  26. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150
  27. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  28. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  29. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  30. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  31. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  32. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  33. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  34. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  35. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  36. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 132
  37. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  38. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124

50 most cited of 2,678 All 2,678 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  2. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't
  3. 2025Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
  4. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  6. 2025Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
  7. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
  8. 2025Token-Efficient RL for LLM Reasoning