mlsanityData to 14 Oct 2026

Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training

arXiv 2309.17179 ↗First posted 29 Sep 2023cs.LG cs.AI cs.CLcited by 295

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Recent works like Tree-of-Thought (ToT) and Reasoning via Planning (RAP) aim to augment the reasoning capabilities of LLMs by using tree-search algorithms to guide multi-step reasoning. These methods rely on prompting a pre-trained model to serve as a value function and focus on problems with low search depth. As a result, these methods will not work in domains where the pre-trained LLM does not have enough knowledge to serve as an effective value function or in domains that require long-horizon planning. To address these limitations, we present an AlphaZero-like tree-search learning framework for LLMs (termed TS-LLM), systematically illustrating how tree-search with a learned value function can guide LLM decoding. TS-LLM distinguishes itself in two key ways. (1) Leveraging a learned value function and AlphaZero-like algorithms, our approach can be generally adaptable to a wide range of tasks, language models of any size, and tasks of varying search depths. (2) Our approach can guide LLMs during both inference and training, iteratively improving the LLM. Empirical results across reasoning, planning, alignment, and decision-making tasks show that TS-LLM outperforms existing approaches and can handle trees with a depth of 64.

Key facts. Cited by 295 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 41 of its 63 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

41 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  2. 2023Large Language Model Guided Tree-of-Thought15 May
  3. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  4. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2023Let's Verify Step by Stepcited by 3,789
25 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  7. 2023LIMA: Less Is More for Alignmentcited by 1,100
  8. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  10. 2022Galactica: A Large Language Model for Sciencecited by 744
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  13. 2023Textbooks Are All You Needcited by 569
  14. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  15. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  16. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  17. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  18. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  19. 2023Planning with Large Language Models for Code Generationcited by 182
  20. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  21. 2021Learning and Planning in Complex Action Spacescited by 77
  22. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 77
  23. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 75
  24. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  25. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Functioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 66; 2025: 167; 2026: 58.

295 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  3. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  4. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  5. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  6. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  7. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  8. 2024Token-Budget-Aware LLM Reasoningcited by 247
  9. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  10. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  11. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  12. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
38 more
  1. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 150
  3. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 144
  4. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  5. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 136
  6. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  7. 2025Process Reward Models That Thinkcited by 118
  8. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  9. 2025Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Designcited by 109
  10. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  11. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 103
  12. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 99
  13. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 97
  14. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  15. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94
  16. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  17. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 88
  18. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 85
  19. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  20. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  21. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  22. 2024o1-Coder: an o1 Replication for Codingcited by 78
  23. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  24. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 74
  25. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learningcited by 74
  26. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tcited by 74
  27. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 70
  28. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
  29. 2025Safety in Large Reasoning Models: A Surveycited by 64
  30. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 63
  31. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 62
  32. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  33. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 58
  34. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 56
  35. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
  36. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 54
  37. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 52
  38. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52

50 most cited of 295 All 295 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition
  2. 2024Enhancing LLM Reasoning with Reward-guided Tree Search
  3. 2026LinTree: Improving LLM Reasoning with Explicitly Structured Search Histories
  4. 2026Revisiting Tree Search for LLMs: Gumbel and Sequential Halving for Budget-Scalable Reasoning
  5. 2025Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey
  6. 2025MITS: Enhanced Tree Search Reasoning for LLMs via Pointwise Mutual Information
  7. 2026Tree of Thoughts as a Classical Heuristic Search Problem: Formal Foundations and Design Patterns
  8. 2025Chain-in-Tree: Back to Sequential Reasoning in LLM Tree Search