Soft Sequence Policy Optimization
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 22 references that are in the map. First posted on arXiv 22 Feb 2026.
Builds on
17 of 22 references.
Closest ancestors · 9
Same direction, newest first
- 2025It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL25 Sep
- 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning25 Sep
- 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
- 2025Geometric-Mean Policy Optimization28 Jul
- 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
- 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2023Let's Verify Step by Step31 May
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 8
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2024Executable Code Actions Elicit Better LLM Agentscited by 564
- 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
- 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
- 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
- 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
- 2025Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learningcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Soft Adaptive Policy Optimization
- 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
- 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
- 2025Group Sequence Policy Optimization
- 2025SSPO: Subsentence-level Policy Optimization
- 2026A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment
- 2025ESPO: Entropy Importance Sampling Policy Optimization
- 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood