mlsanityData to 7 Oct 2026

CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models

arXiv 2505.12504 ↗First posted 18 May 2025cs.LG cs.AIcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in rule-based reinforcement learning (RL) have significantly improved the reasoning capability of language models (LMs) with rule-based rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO -- often suffer from training instability, where large policy updates and improper clipping can lead to training collapse. To address this issue, we propose Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel algorithm designed to stabilize policy learning in LMs. CPGD introduces a policy drift constraint based on KL divergence to dynamically regularize policy updates, and leverages a clip mechanism on the logarithm of the ratio to prevent excessive policy updates. We provide theoretical justification for CPGD and demonstrate through empirical analysis that it mitigates the instability observed in prior approaches. Furthermore, we show that CPGD significantly improves performance while maintaining training stability. Our implementation balances theoretical rigor with practical usability, offering a robust alternative for RL in the post-training of LMs. We release our code at https://github.com/ModalMinds/MM-EUREKA.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 35 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

17 of 35 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  4. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  5. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  6. 2019A Theory of Regularized Markov Decision Processescited by 279
  7. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  8. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  9. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  10. 2024Learning Dynamics of LLM Finetuningcited by 93
  11. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 49
  3. 2025VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMscited by 6
  4. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policycited by 6
  5. 2025Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Modelscited by 3
  6. 2025DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Trainingcited by 1
  7. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
  8. 2025COSMO-RL: Towards Trustworthy LMRMs via Joint Safety and Stabilitycited by 0
  9. 2025One4Many-StablePacker: An Efficient Deep Reinforcement Learning Framework for the 3D Bin Packing Problemcited by 0
  10. 2026DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Trainingcited by 0
  11. 2026Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  2. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
  3. 2026Stabilizing Policy Optimization via Logits Convexity
  4. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
  5. 2025COPO: Consistency-Aware Policy Optimization
  6. 2026Clipping-Free Policy Optimization for Large Language Models
  7. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models
  8. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight