mlsanityData to 14 Oct 2026

TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling

arXiv 2508.17445 ↗First posted 24 Aug 2025cs.LG cs.CLcited by 61

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in aligning large language models via reinforcement learning have achieved remarkable gains in solving complex reasoning problems, but at the cost of expensive on-policy rollouts and limited exploration of diverse reasoning paths. In this work, we introduce TreePO, involving a self-guided rollout algorithm that views sequence generation as a tree-structured searching process. Composed of dynamic tree sampling policy and fixed-length segment decoding, TreePO leverages local uncertainty to warrant additional branches. By amortizing computation across common prefixes and pruning low-value paths early, TreePO essentially reduces the per-update compute burden while preserving or enhancing exploration diversity. Key contributions include: (1) a segment-wise sampling algorithm that alleviates the KV cache burden through contiguous segments and spawns new branches along with an early-stop mechanism; (2) a tree-based segment-level advantage estimation that considers both global and local proximal policy optimization. and (3) analysis on the effectiveness of probability and quality-driven dynamic divergence and fallback strategy. We empirically validate the performance gain of TreePO on a set reasoning benchmarks and the efficiency saving of GPU hours from 22% up to 43% of the sampling design for the trained models, meanwhile showing up to 40% reduction at trajectory-level and 35% at token-level sampling compute for the existing models. While offering a free lunch of inference efficiency, TreePO reveals a practical path toward scaling RL-based post-training with fewer samples and less compute. Home page locates at https://m-a-p.ai/TreePO.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 25 references that are in the map. First posted on arXiv 24 Aug 2025.

Builds on

18 of 25 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models28 Jun
  2. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 14

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  9. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  10. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 161
  11. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  12. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modelingcited by 21
2 more
  1. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 13
  2. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 39.

61 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  4. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  5. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  6. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  7. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
  8. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  9. 2025Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocationcited by 22
  10. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 19
  11. 2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewardscited by 19
  12. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
38 more
  1. 2025ParallelMuse: Agentic Parallel Thinking for Deep Information Seekingcited by 14
  2. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Modelscited by 13
  3. 2025Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewardscited by 13
  4. 2025Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoningcited by 12
  5. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 11
  6. 2026Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learningcited by 11
  7. 2025Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPOcited by 10
  8. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 10
  9. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 9
  10. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  11. 2025Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attackscited by 8
  12. 2026Reinforced Efficient Reasoning via Semantically Diverse Explorationcited by 8
  13. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 8
  14. 2026ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialoguecited by 8
  15. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 8
  16. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  17. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Modelscited by 6
  18. 2025Teaching Language Models to Reason with Toolscited by 5
  19. 2026TreeAdv: Tree-Structured Advantage Redistribution for Group-Based RLcited by 5
  20. 2026TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAGcited by 5
  21. 2026ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibrationcited by 4
  22. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  23. 2026TopoCurate:Modeling Interaction Topology for Tool-Use Agent Trainingcited by 4
  24. 2025DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generationcited by 3
  25. 2026Entropy-Tree: Tree-Based Decoding with Entropy-Guided Explorationcited by 3
  26. 2026Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Treescited by 2
  27. 2026RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimizationcited by 2
  28. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  29. 2026Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routingcited by 2
  30. 2026BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?cited by 2
  31. 2026Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVRcited by 2
  32. 2026On the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample Efficiencycited by 1
  33. 2026PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoningcited by 1
  34. 2026MARS2: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generationcited by 1
  35. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  36. 2026Tree-based Credit Assignment for Multi-Agent Memory Systemcited by 1
  37. 2026Confidence-Aware Alignment Makes Reasoning LLMs More Reliablecited by 1
  38. 2026Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimationcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CATPO: Critique-Augmented Tree Policy Optimization
  2. 2025TreeRPO: Tree Relative Policy Optimization
  3. 2026GraphPO: Graph-based Policy Optimization for Reasoning Models
  4. 2026Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents
  5. 2025Policy Guided Tree Search for Enhanced LLM Reasoning
  6. 2025SPOT: Scalable Policy Optimization with Trees for Markov Decision Processes
  7. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
  8. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoning