mlsanityData to 14 Oct 2026

From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight

arXiv 2603.14389 ↗First posted 15 Mar 2026cs.LG cs.AIcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed a leap in Large Language Model (LLM) reasoning, yet its optimization dynamics remain fragile. Standard algorithms like GRPO enforce stability via "hard clipping", which inadvertently stifles exploration by discarding gradients of tokens outside the trust region. While recent "soft clipping" methods attempt to recover these gradients, they suffer from a critical challenge: relying on log-probability gradient (nablaθlog πθ) yields divergent weights as probabilities vanish, destabilizing LLM training. We rethink this convention by establishing probability gradient (nablaθ πθ) as the superior optimization primitive. Accordingly, we propose Decoupled Gradient Policy Optimization (DGPO), which employs a decoupled decay mechanism based on importance sampling ratios. By applying asymmetric, continuous decay to boundary tokens, DGPO resolves the conflict between stability and sustained exploration. Extensive experiments across DeepSeek-R1-Distill-Qwen series models (1.5B/7B/14B) demonstrate that DGPO consistently outperforms strong baselines on various mathematical benchmarks, offering a robust and scalable solution for RLVR. Our code and implementation are available at: https://github.com/FlyTune/DGPO-RL.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 42 references that are in the map. First posted on arXiv 15 Mar 2026.

Builds on

28 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning25 Sep
  2. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization11 Aug
  3. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
  4. 2023Let's Verify Step by Step31 May
  5. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  8. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
  9. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  10. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  11. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)cited by 38
  12. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
10 more
  1. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 25
  2. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  3. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  4. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
  5. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  6. 2025Who Stole Your Data? A Method for Detecting Unauthorized RAG Theftcited by 15
  7. 2026Automated Optimization Modeling via a Localizable Error-Driven Perspectivecited by 14
  8. 2025Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Naturecited by 11
  9. 2026Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearningcited by 10
  10. 2024Narrative Player: Reviving Data Narratives with Visualscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Group-Reflective Self-Distillation for Agentic Reinforcement Learningcited by 6
  2. 2026Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectoriescited by 4
  3. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  4. 2026Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generationcited by 1
  5. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0
  6. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL
  2. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  3. 2026UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma
  4. 2026Clipping-Free Policy Optimization for Large Language Models
  5. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  6. 2026A Unified Framework for Rethinking Policy Divergence Measures in GRPO
  7. 2026Rethinking the Divergence Regularization in LLM RL
  8. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning