mlsanityData to 7 Oct 2026

AT2PO: Agentic Turn-based Policy Optimization via Tree Search

arXiv 2601.04767 ↗First posted 8 Jan 2026cs.AI cs.CLcited by 7

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

LLM agents have emerged as powerful systems for tackling multi-turn tasks by interleaving internal reasoning and external tool interactions. Agentic Reinforcement Learning has recently drawn significant research attention as a critical post-training paradigm to further refine these capabilities. In this paper, we present AT2PO (Agentic Turn-based Policy Optimization via Tree Search), a unified framework for multi-turn agentic RL that addresses three core challenges: limited exploration diversity, sparse credit assignment, and misaligned policy optimization. AT2PO introduces a turn-level tree structure that jointly enables Entropy-Guided Tree Expansion for strategic exploration and Turn-wise Credit Assignment for fine-grained reward propagation from sparse outcomes. Complementing this, we propose Agentic Turn-based Policy Optimization, a turn-level learning objective that aligns policy updates with the natural decision granularity of agentic interactions. ATPO is orthogonal to tree search and can be readily integrated into any multi-turn RL pipeline. Experiments across seven benchmarks demonstrate consistent improvements over the state-of-the-art baseline by up to 1.84 percentage points in average, with ablation studies validating the effectiveness of each component. Our code is available at https://github.com/zzfoutofspace/ATPO.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 33 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

23 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
9 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  2. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  3. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
  4. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  5. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 75
  6. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  7. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
  8. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learningcited by 23
  9. 2024SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibrationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimizationcited by 5
  2. 2026TCPO: Turn-Level Credit Policy Optimizationcited by 2
  3. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
  4. 2026Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Contextcited by 1
  5. 2026SIPO: Selective-Inference Policy Optimization for Tree-Structured Agentic RLcited by 0
  6. 2026ASCT: Attentive Search over Counterfactual Trees for Credit Assignment in Agentic Reinforcement Learningcited by 0
  7. 2026OPTS-TTPO: Enhancing Finite-Sample Policy-Gradient Learning with Tree Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
  2. 2026APPO: Agentic Procedural Policy Optimization
  3. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment
  4. 2026A2TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
  5. 2026Process Reward Informed Tree Rollout for Effective Multi-Turn RL
  6. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
  7. 2025Tree Search for LLM Agent Reinforcement Learning
  8. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning