mlsanityData to 14 Oct 2026

SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

arXiv 2608.24870 ↗First posted 25 Aug 2026cs.AIcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 25 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

23 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning20 Aug
  2. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning8 Jul
  3. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction12 May

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  6. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  7. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  8. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 98
  9. 2026Maximum Likelihood Reinforcement Learningcited by 36
  10. 2025Single-stream Policy Optimizationcited by 28
  11. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 12
  12. 2025Stable and Efficient Single-Rollout RL for Multimodal Reasoningcited by 8
8 more
  1. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  2. 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rolloutscited by 5
  3. 2026Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learningcited by 5
  4. 2026VIMPO: Value-Implicit Policy Optimization for LLMscited by 4
  5. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPOcited by 3
  6. 2026SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoningcited by 2
  7. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
  8. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agentscited by 0
  2. 2026Long-Horizon RL Needs Muoncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning with Decomposed Subtasks
  2. 2026ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning
  3. 2026Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
  4. 2025Dual Alignment Maximin Optimization for Offline Model-based RL
  5. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
  6. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  7. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
  8. 2026Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning