Smooth Gate Functions for Soft Advantage Policy Optimization
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Group Relative Policy Optimization (GRPO) has significantly advanced the training of large language models and enhanced their reasoning capabilities, while it remains susceptible to instability due to the use of hard clipping. Soft Adaptive Policy Optimization (SAPO) addresses this limitation by replacing clipping with a smooth sigmoid-based gate function, which leads to more stable updates. We have decided to push this theory further and investigate the impact of different gate functions on both training stability and final model performance. We formalize the key properties that admissible gates should satisfy and identify several families of such functions for empirical evaluation. This paper presents an analysis of our findings based on experiments conducted with the Qwen2.5-7B-Instruct model on mathematical reasoning tasks. These results provide practical guidance for designing smoother and more robust policy optimization objectives for large language model training.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 18 references that are in the map. First posted on arXiv 22 Feb 2026.
Builds on
13 of 18 references.
Closest ancestors · 4
Same direction, newest first
- 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning25 Sep
- 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
- 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 9
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2015Trust Region Policy Optimizationcited by 2,503
- 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 30
- 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
- 2022You May Not Need Ratio Clipping in PPOcited by 19
- 2019Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learningcited by 12
- 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Soft Adaptive Policy Optimization
- 2026Soft Sequence Policy Optimization
- 2026GRPO Training Dynamics for Small Language Models
- 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
- 2025Geometric-Mean Policy Optimization
- 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
- 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
- 2025COPO: Consistency-Aware Policy Optimization