mlsanityData to 7 Oct 2026

Single-stream Policy Optimization

arXiv 2509.13232 ↗First posted 16 Sep 2025cs.LG cs.AI stat.MLcited by 27

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We revisit policy-gradient optimization for Large Language Models (LLMs) from a single-stream perspective. Prevailing group-based methods like GRPO reduce variance with on-the-fly baselines but suffer from critical flaws: frequent degenerate groups erase learning signals, and synchronization barriers hinder scalability. We introduce Single-stream Policy Optimization (SPO), which eliminates these issues by design. SPO replaces per-group baselines with a persistent, KL-adaptive value tracker and normalizes advantages globally across the batch, providing a stable, low-variance learning signal for every sample. Being group-free, SPO enables higher throughput and scales effectively in long-horizon or tool-integrated settings where generation times vary. Furthermore, the persistent value tracker naturally enables an adaptive curriculum via prioritized sampling. Experiments using Qwen3-8B show that SPO converges more smoothly and attains higher accuracy than GRPO, while eliminating computation wasted on degenerate groups. Ablation studies confirm that SPO's gains stem from its principled approach to baseline estimation and advantage normalization, offering a more robust and efficient path for LLM reasoning. Across five hard math benchmarks with Qwen3 8B, SPO improves the average maj@32 by +3.4 percentage points (pp) over GRPO, driven by substantial absolute point gains on challenging datasets, including +7.3 pp on BRUMO 25, +4.4 pp on AIME 25, +3.3 pp on HMMT 25, and achieves consistent relative gain in pass@k across the evaluated k values. SPO's success challenges the prevailing trend of adding incidental complexity to RL algorithms, highlighting a path where fundamental principles, not architectural workarounds, drive the next wave of progress in LLM reasoning.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 44 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

37 of 44 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025rStar2-Agent: Agentic Reasoning Technical Report28 Aug
  2. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
  3. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  4. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  5. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy16 Jun
  6. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  7. 2025On-Policy RL with Optimal Reward Baseline29 May
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  9. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression27 May
  10. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
3 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2025Group Sequence Policy Optimizationcited by 788
  9. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  10. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  11. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  12. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
10 more
  1. 2025Deep Think with Confidencecited by 199
  2. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  3. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  4. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  5. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  6. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  7. 2025Understanding Tool-Integrated Reasoningcited by 38
  8. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  9. 2024InfAlign: Inference-aware language model alignmentcited by 34
  10. 2025Agents Play Thousands of 3D Video Gamescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 21.

27 papers.

Sort:

By area:

  1. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 26
  2. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  3. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  4. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  5. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 11
  6. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  7. 2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
  8. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPOcited by 9
  9. 2025Stable and Efficient Single-Rollout RL for Multimodal Reasoningcited by 8
  10. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  11. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 7
  12. 2026RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Modelscited by 3
15 more
  1. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
  2. 2026Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Workscited by 3
  3. 2026Best Practice Critic Optimizationcited by 3
  4. 2026Policy Improvement Reinforcement Learningcited by 2
  5. 2026Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolutioncited by 2
  6. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  7. 2026Co-Evolution of Policy and Internal Reward for Language Agentscited by 1
  8. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  9. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
  10. 2026UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrationscited by 1
  11. 2026SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RLcited by 1
  12. 2026Decoupling Strategy and Execution in Task-Focused Dialogue via Goal-Oriented Preference Optimizationcited by 0
  13. 2026Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Maskingcited by 0
  14. 2026Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signalscited by 0
  15. 2026Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation
  2. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoning
  3. 2026Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
  4. 2026V0: A Generalist Value Model for Any Policy at State Zero
  5. 2025Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
  6. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
  7. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  8. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood