mlsanityData to 7 Oct 2026

Enhancing LLM Reasoning with Reward-guided Tree Search

arXiv 2411.11694 ↗First posted 18 Nov 2024cs.CL cs.AIcited by 58

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Recently, test-time scaling has garnered significant attention from the research community, largely due to the substantial advancements of the o1 model released by OpenAI. By allocating more computational resources during the inference phase, large language models~(LLMs) can extensively explore the solution space by generating more thought tokens or diverse solutions, thereby producing more accurate responses. However, developing an o1-like reasoning approach is challenging, and researchers have been making various attempts to advance this open area of research. In this paper, we present a preliminary exploration into enhancing the reasoning abilities of LLMs through reward-guided tree search algorithms. This framework is implemented by integrating the policy model, reward model, and search algorithm. It is primarily constructed around a tree search algorithm, where the policy model navigates a dynamically expanding tree guided by a specially trained reward model. The implemented framework is denoted as STILL-1. We thoroughly explore various design considerations necessary for implementing this framework and provide a detailed report of the technical aspects. To assess the effectiveness of our approach, we focus on mathematical reasoning tasks and conduct extensive evaluations on four challenging datasets, significantly enhancing the reasoning abilities of LLMs.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 0 of its 45 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

0 of 45 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 46; 2026: 8.

58 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  4. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  5. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  6. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  7. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  8. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  9. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  10. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  11. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillationcited by 39
  12. 2025Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoningcited by 36
38 more
  1. 2025O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoningcited by 34
  2. 2025Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinkingcited by 29
  3. 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoningcited by 28
  4. 2025DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learningcited by 23
  5. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  6. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  7. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
  8. 2025Accelerating Large Language Model Reasoning via Speculative Searchcited by 16
  9. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVRcited by 16
  10. 2025Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scalingcited by 15
  11. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  12. 2025Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughtscited by 12
  13. 2025CoAT: Chain-of-Associated-Thoughts Framework for Enhancing Large Language Models Reasoningcited by 11
  14. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  15. 2025RARE: Retrieval-Augmented Reasoning Modelingcited by 11
  16. 2025Slow Thinking for Sequential Recommendationcited by 11
  17. 2025Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoningcited by 11
  18. 2025RoT: Enhancing Table Reasoning with Iterative Row-Wise Traversalscited by 10
  19. 2025VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanismcited by 7
  20. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  21. 2025Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Modelscited by 6
  22. 2024Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertisecited by 5
  23. 2024YuLan-Mini: An Open Data-efficient Language Modelcited by 5
  24. 2025FANS -- Formal Answer Selection for Natural Language Math Reasoning Using Lean4cited by 5
  25. 2025From Chaos to Order: The Atomic Reasoner Framework for Fine-grained Reasoning in Large Language Modelscited by 5
  26. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  27. 2025LLMPC: Large Language Model Predictive Controlcited by 4
  28. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decodingcited by 4
  29. 2025Decoupling Knowledge and Reasoning in LLMs: An Exploration Using Cognitive Dual-System Theorycited by 4
  30. 2025Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thoughtcited by 4
  31. 2025More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoningcited by 4
  32. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  33. 2025Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Designcited by 3
  34. 2025Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Frameworkcited by 2
  35. 2025EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruningcited by 1
  36. 2025Pedagogy-R1: Pedagogically-Aligned Reasoning Model with Balanced Educational Benchmarkcited by 1
  37. 2025Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorationscited by 1
  38. 2025Stepwise Reasoning Checkpoint Analysis: A Test Time Scaling Method to Enhance LLMs' Reasoningcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey
  2. 2025MITS: Enhanced Tree Search Reasoning for LLMs via Pointwise Mutual Information
  3. 2025Adaptive Test-Time Reasoning via Reward-Guided Dual-Phase Search
  4. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
  5. 2026LinTree: Improving LLM Reasoning with Explicitly Structured Search Histories
  6. 2025Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling
  7. 2025Policy Guided Tree Search for Enhanced LLM Reasoning
  8. 2025SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition