mlsanityData to 14 Oct 2026

Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs

arXiv 2512.17008 ↗First posted 18 Dec 2025cs.LGcited by 30

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. However, directly applying the widely used Group Relative Policy Optimization (GRPO) algorithm to multi-turn tasks exposes notable limitations, particularly in scenarios requiring long-horizon reasoning. To address these challenges, we investigate more stable and effective advantage estimation strategies, especially for multi-turn settings. We first explore Proximal Policy Optimization (PPO) as an alternative and find it to be more robust than GRPO. To further enhance PPO in multi-turn scenarios, we introduce turn-PPO, a variant that operates on a turn-level MDP formulation, as opposed to the commonly used token-level MDP. Our results on the WebShop and Sokoban datasets demonstrate the effectiveness of turn-PPO, both with and without long reasoning components.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 33 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

18 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  3. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  8. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  9. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  10. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  11. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  12. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
3 more
  1. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 98
  2. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  3. 2025PIPA: Preference Alignment as Prior-Informed Statistical Estimationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 30.

30 papers.

Sort:

By area:

  1. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 23
  2. 2026GAGPO: Generalized Advantage Grouped Policy Optimizationcited by 9
  3. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  4. 2026Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibrationcited by 7
  5. 2026Retrospective Progress-Aware Self-Refinement for LLM Agent Trainingcited by 6
  6. 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rolloutscited by 5
  7. 2026ProAct: Agentic Lookahead in Interactive Environmentscited by 3
  8. 2026PRIME: Training Free Proactive Reasoning via Iterative Memory Evolution for User-Centric Agentcited by 3
  9. 2026CAST: Game Solvers as Turn-Level Teachers for LLM Agentscited by 3
  10. 2026TCPO: Turn-Level Credit Policy Optimizationcited by 3
  11. 2026Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agentscited by 3
  12. 2026How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learningcited by 3
18 more
  1. 2026Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairingcited by 2
  2. 2026SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Trainingcited by 2
  3. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  4. 2026Drowning in Routine: Signal Dilution in Multi-Turn Agent Trainingcited by 1
  5. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agentscited by 1
  6. 2026SkillGate: Training In-Policy Skill Selection in Long-Horizon Agentscited by 1
  7. 2026ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RLcited by 1
  8. 2026Can the Environment Speak for Itself? T2-GRPO: A Turn-Trajectory Group Relative Policy Optimization for Caregiver Agentscited by 0
  9. 2026Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproductioncited by 0
  10. 2026MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agentscited by 0
  11. 2026Harness-G: A Graph-Structured Harness for Search Agentscited by 0
  12. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0
  13. 2026Toolcompass: Guiding Tool Trialing, Not Suppressing Itcited by 0
  14. 2026ProCredit: From Outcome Rewards to Progress Credit in Agentic Reinforcement Learningcited by 0
  15. 2026Coding Agent Memory Post-training: Unlocking the Memory Potential of Pre-trained File Operations for Long-Horizon Tasks via Reinforcement Learningcited by 0
  16. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learningcited by 0
  17. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agentscited by 0
  18. 2026C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SeeUPO: Sequence-Level Agentic-RL with Convergence Guarantees
  2. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment
  3. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
  4. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  5. 2026PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks
  6. 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Search
  7. 2025MAPO: Mixed Advantage Policy Optimization
  8. 2026Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence