mlsanityData to 14 Oct 2026

DCPO: Dynamic Clipping Policy Optimization

arXiv 2509.02333 ↗First posted 2 Sep 2025cs.CL cs.AI cs.LGcited by 40

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising framework for enhancing the reasoning capabilities of large language models. However, existing approaches such as GRPO often suffer from zero gradients. This problem arises primarily due to fixed clipping bounds for token-level probability ratios and the standardization of identical rewards, which can lead to ineffective gradient updates and underutilization of generated responses. In this work, we propose Dynamic Clipping Policy Optimization(DCPO), which introduces a dynamic clipping strategy that adaptively adjusts clipping bounds based on token-specific prior probabilities to enhance token-level exploration, and a smooth advantage standardization technique that standardizes rewards across cumulative training steps to improve the response-level effective utilization of generated responses. DCPO achieved state-of-the-art performance on four benchmarks based on four different models. In particular, DCPO achieved an Avg@1 of 46.7 under greedy decoding and an Avg@32 of 38.8 under 32 times sampling on the AIME24 benchmark, surpassing DAPO (36.7/31.6), GRPO (36.7/32.1) and GSPO (40.0/34.9) on the Qwen2.5-Math-7B model. On the AIME25 benchmark based on Qwen2.5-14B, DCPO achieves a performance of (23.3/19.0), surpassing GRPO (13.3/10.5), DAPO (20.0/15.3) and GSPO (16.7/9.9). Furthermore, DCPO achieved an average 28% improvement in the nonzero advantage over GRPO in four models, doubled the training efficiency over DAPO, and significantly reduced the token clipping ratio by an order of magnitude compared to both GRPO and DAPO, while achieving superior performance. These results highlight DCPO's effectiveness in leveraging generated data more efficiently for reinforcement learning in large language models.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 18 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

16 of 18 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2025Group Sequence Policy Optimizationcited by 835
  9. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 28.

40 papers.

Sort:

By area:

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  2. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  3. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  4. 2025VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillationcited by 31
  5. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Modelscited by 27
  6. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
  7. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 15
  8. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 11
  9. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 11
  10. 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoningcited by 11
  11. 2025ESPO: Entropy Importance Sampling Policy Optimizationcited by 9
  12. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
28 more
  1. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimizationcited by 7
  2. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  3. 2026Revisiting DAgger in the Era of LLM-Agentscited by 7
  4. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  5. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoningcited by 5
  6. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?cited by 5
  7. 2025SSPO: Subsentence-level Policy Optimizationcited by 5
  8. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Modelscited by 5
  9. 2025Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learningcited by 4
  10. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 3
  11. 2026Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimizationcited by 3
  12. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilitiescited by 2
  13. 2026Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generationcited by 2
  14. 2026Policy Improvement Reinforcement Learningcited by 2
  15. 2026Online Causal Kalman Filtering for Stable and Effective Policy Optimizationcited by 1
  16. 2026Stabilizing Policy Optimization via Logits Convexitycited by 1
  17. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  18. 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Modelscited by 1
  19. 2026Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distancecited by 1
  20. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  21. 2026Baichuan-M4: A Clinical-Grade Medical Agent System for Continuous Carecited by 1
  22. 2026GUI-AC: Enhancing Continual Learning in GUI Agentscited by 1
  23. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  24. 2026A Unified Framework for Rethinking Policy Divergence Measures in GRPOcited by 0
  25. 2026Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellationcited by 0
  26. 2026Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learningcited by 0
  27. 2026Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergencecited by 0
  28. 2026Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR
  2. 2026What are Key Factors for Updates in RL for LLM Reasoning?
  3. 2026Clipping-Free Policy Optimization for Large Language Models
  4. 2026Towards Better Training Signal: Advantage Clipped Policy Optimization
  5. 2026A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization
  6. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
  7. 2026All-Quadrant Bounded Clipping GRPO: Closing the Unbounded Blind Spot for Stable and Generalizable Training
  8. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective