mlsanityData to 14 Oct 2026

TreeRL: LLM Reinforcement Learning with On-Policy Tree Search

arXiv 2506.11902 ↗First posted 13 Jun 2025cs.LG cs.CLcited by 80

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Reinforcement learning (RL) with tree search has demonstrated superior performance in traditional reasoning tasks. Compared to conventional independent chain sampling strategies with outcome supervision, tree search enables better exploration of the reasoning space and provides dense, on-policy process rewards during RL training but remains under-explored in On-Policy LLM RL. We propose TreeRL, a reinforcement learning framework that directly incorporates on-policy tree search for RL training. Our approach includes intermediate supervision and eliminates the need for a separate reward model training. Existing approaches typically train a separate process reward model, which can suffer from distribution mismatch and reward hacking. We also introduce a cost-effective tree search approach that achieves higher search efficiency under the same generation token budget by strategically branching from high-uncertainty intermediate steps rather than using random branching. Experiments on challenging math and code reasoning benchmarks demonstrate that TreeRL achieves superior performance compared to traditional ChainRL, highlighting the potential of tree search for LLM. TreeRL is open-sourced at https://github.com/THUDM/TreeRL.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 42 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

31 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  2. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning1 May
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
16 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  7. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  8. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  9. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  10. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  11. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  12. 2023Large Language Model Guided Tree-of-Thoughtcited by 221
  13. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  14. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  15. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  16. 2023The Role of Baselines in Policy Gradient Optimizationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 60.

80 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  3. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 61
  4. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  5. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 36
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  7. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
  8. 2025Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocationcited by 22
  9. 2025Rethinking Thinking Tokens: LLMs as Improvement Operatorscited by 21
  10. 2026HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systemscited by 21
  11. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 18
  12. 2025ParallelMuse: Agentic Parallel Thinking for Deep Information Seekingcited by 14
38 more
  1. 2026CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agentscited by 14
  2. 2025Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewardscited by 13
  3. 2026Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoningcited by 12
  4. 2025Repurposing Synthetic Data for Fine-grained Search Agent Supervisioncited by 11
  5. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 11
  6. 2026Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learningcited by 11
  7. 2025GRPO is Secretly a Process Reward Modelcited by 10
  8. 2025Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPOcited by 10
  9. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 10
  10. 2026PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planningcited by 9
  11. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  12. 2026Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoningcited by 9
  13. 2025Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attackscited by 8
  14. 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Searchcited by 8
  15. 2026ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialoguecited by 8
  16. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
  17. 2025CARL: Criticality-Aware Agentic Reinforcement Learningcited by 5
  18. 2026TreeAdv: Tree-Structured Advantage Redistribution for Group-Based RLcited by 5
  19. 2026TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAGcited by 5
  20. 2026From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoningcited by 5
  21. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  22. 2026Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoningcited by 4
  23. 2026Self-Improving Language Models with Bidirectional Evolutionary Searchcited by 4
  24. 2026APPO: Agentic Procedural Policy Optimizationcited by 4
  25. 2026Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimizationcited by 4
  26. 2026Entropy-Tree: Tree-Based Decoding with Entropy-Guided Explorationcited by 3
  27. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMscited by 3
  28. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMscited by 3
  29. 2026GraphPO: Graph-based Policy Optimization for Reasoning Modelscited by 3
  30. 2025The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Samplingcited by 2
  31. 2026BFS-PO: Best-First Search for Large Reasoning Modelscited by 2
  32. 2026A2TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clippingcited by 2
  33. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  34. 2026Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modelingcited by 2
  35. 2026Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routingcited by 2
  36. 2026MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agentscited by 2
  37. 2026Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVRcited by 2
  38. 2026Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flatteningcited by 2

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Policy Guided Tree Search for Enhanced LLM Reasoning
  2. 2025TreeRPO: Tree Relative Policy Optimization
  3. 2025TTRL: Test-Time Reinforcement Learning
  4. 2026CATPO: Critique-Augmented Tree Policy Optimization
  5. 2024Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization
  6. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Trees
  7. 2025Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey
  8. 2024Enhancing LLM Reasoning with Reward-guided Tree Search