mlsanityData to 14 Oct 2026

Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning

arXiv 2512.05591 ↗First posted 5 Dec 2025cs.LG cs.CLcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Large language model post-training relies on reinforcement learning to improve model capability and alignment quality. However, the off-policy training paradigm introduces distribution shift, which often pushes the policy beyond the trust region, leading to training instabilities manifested as fluctuations in policy entropy and unstable gradients. Although PPO-Clip mitigates this issue through importance clipping, it still overlooks the global distributional shift of actions. To address these challenges, we propose using the entropy ratio between the current and previous policies as a new global metric that effectively quantifies the relative change in policy exploration throughout updates. Building on this metric, we introduce an Entropy Ratio Clipping (ERC) mechanism that imposes bidirectional constraints on the entropy ratio. This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions. We integrate ERC into both DAPO and GPPO reinforcement learning algorithms. Experiments across multiple benchmarks show that ERC consistently improves performance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 17 references that are in the map. First posted on arXiv 5 Dec 2025.

Builds on

12 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning25 Sep

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  8. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  9. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  10. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
  11. 2025Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignmentcited by 6
  2. 2026Soft Sequence Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
  2. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
  3. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective
  4. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
  5. 2026Entropy-Preserving Reinforcement Learning
  6. 2025ESPO: Entropy Importance Sampling Policy Optimization
  7. 2025Non-Asymptotic Global Convergence of PPO-Clip
  8. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization