mlsanityData to 14 Oct 2026

It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL

arXiv 2509.21282 ↗First posted 25 Sep 2025cs.LG cs.AIcited by 7

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Training large language models (LLMs) with reinforcement learning (RL) methods such as PPO and GRPO commonly relies on ratio clipping to stabilise updates. While effective at preventing instability, clipping discards information, introduces gradient discontinuities and can prevent exploration of better policies. Inspired by label smoothing, we propose Probability Smoothing Policy Optimisation (PSPO). PSPO smooths current policy probabilities toward the behaviour policy before computing importance ratios, creating a soft trust region that preserves gradients while preventing destabilising updates. Unlike prior soft clipping approaches that use sigmoid-based transformations which can suffer from vanishing gradients and saturation, our method uses a linear interpolation, providing simpler and more robust gradient preservation. Empirically, GR-PSPO outperforms clipping and sigmoid-based alternatives on mathematical reasoning benchmarks when refining models with prior domain knowledge, achieving an accuracy of 79.9% on GSM8K and 59.6% on MATH for Qwen2-Math-1.5B.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 37 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

28 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMs3 Jul
  2. 2023Let's Verify Step by Step31 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2019When Does Label Smoothing Help?cited by 988
13 more
  1. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  2. 2025Group Sequence Policy Optimizationcited by 835
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  5. 2022Large Language Models Are Reasoning Teacherscited by 329
  6. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  7. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  8. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  9. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  10. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  11. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  12. 2022You May Not Need Ratio Clipping in PPOcited by 19
  13. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signalscited by 1
  2. 2026Soft Sequence Policy Optimizationcited by 0
  3. 2026KLip-PPO: A per-sample KL perspective on PPO-Clipcited by 0
  4. 2026DocHRL: A Hierarchical Reinforcement Learning Framework for Cost-Optimised Document Classificationcited by 0
  5. 2026Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysiscited by 0
  6. 2026A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clippingcited by 0
  7. 2026Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
  2. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
  3. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
  4. 2026Rethinking the Divergence Regularization in LLM RL
  5. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  6. 2026BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
  7. 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning
  8. 2025Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning