mlsanityData to 7 Oct 2026

SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

arXiv 2604.08865 ↗First posted 10 Apr 2026cs.AIcited by 7

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 48 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

34 of 48 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay17 Mar
  2. 2026How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization22 Feb
  3. 2025AGRO-SQL: Agentic Group-Relative Optimization with High-Fidelity Data Synthesis29 Dec
  4. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation7 Dec
  5. 2025Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards18 Oct
  6. 2025NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problems18 Oct
  7. 2025Geometric-Mean Policy Optimization28 Jul
  8. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
  9. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning20 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
3 more
  1. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  2. 2023Let's Verify Step by Step31 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  5. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  6. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 45
  7. 2025Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Trainingcited by 42
  8. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  9. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  10. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuningcited by 20
  11. 2026Proximity-Based Multi-Turn Optimization: Practical Credit Assignment for LLM Agent Trainingcited by 18
  12. 2026Verified Critical Step Optimization for LLM Agentscited by 13
7 more
  1. 2024How Do Humans Write Code? Large Models Do It the Same Way Toocited by 11
  2. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Taskscited by 10
  3. 2025ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoningcited by 10
  4. 2026No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learningcited by 7
  5. 2026Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectificationcited by 5
  6. 2026InCoder-32B-Thinking: Industrial Code World Model for Thinkingcited by 4
  7. 2026SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal Statescited by 2
  2. 2026Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMscited by 1
  3. 2026Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)cited by 1
  4. 2026Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flatteningcited by 1
  5. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0
  6. 2026GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Modelscited by 0
  7. 2026MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Truncated Proximal Policy Optimization
  2. 2026LamPO: A Lambda Style Policy Optimization for Reasoning Language Models
  3. 2025HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs
  4. 2026Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning
  5. 2025GPO: Learning from Critical Steps to Improve LLM Reasoning
  6. 2025Optimizing Anytime Reasoning via Budget Relative Policy Optimization
  7. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  8. 2025Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement