mlsanityData to 14 Oct 2026

Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

arXiv 2406.14283 ↗First posted 20 Jun 2024cs.AIcited by 97

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive capability in many natural language tasks. However, the auto-regressive generation process makes LLMs prone to produce errors, hallucinations and inconsistent statements when performing multi-step reasoning. In this paper, by casting multi-step reasoning of LLMs as a heuristic search problem, we aim to alleviate the pathology by introducing Q*, a general, versatile and agile framework for guiding LLMs decoding process with deliberative planning. By learning a plug-and-play Q-value model as heuristic function for estimating expected future rewards, our Q* can effectively guide LLMs to select the most promising next reasoning step without fine-tuning LLMs for the current task, which avoids the significant computational overhead and potential risk of performance degeneration on other tasks. Extensive experiments on GSM8K, MATH and MBPP demonstrate the superiority of our method, contributing to improving the reasoning performance of existing open-source LLMs.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 52 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

38 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Code Llama: Open Foundation Models for Code24 Aug

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  11. 2023Let's Verify Step by Stepcited by 3,789
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
25 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  4. 2023Teaching Large Language Models to Self-Debugcited by 988
  5. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  8. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  10. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  13. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  14. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  15. 2023Llemma: An Open Language Model For Mathematicscited by 388
  16. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  17. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  18. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  19. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 207
  20. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  21. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  22. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 105
  23. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 103
  24. 2023SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledgecited by 49
  25. 2023The ART of LLM Refinement: Ask, Refine, and Trustcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 59; 2026: 20.

97 papers.

Sort:

By area:

  1. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  3. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  4. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 169
  5. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  6. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  7. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 135
  8. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  9. 2024Process Reward Model with Q-Value Rankingscited by 80
  10. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  11. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  12. 2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Makingcited by 42
38 more
  1. 2024RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinementcited by 40
  2. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  3. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 39
  4. 2024Benchmarking Agentic Workflow Generationcited by 37
  5. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 37
  6. 2024HunyuanProver: A Scalable Data Synthesis Framework and Guided Tree Search for Automated Theorem Provingcited by 33
  7. 2024LiteSearch: Efficacious Tree Search for LLMcited by 31
  8. 2025Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinkingcited by 31
  9. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 29
  10. 2025AgentRM: Enhancing Agent Generalization with Reward Modelingcited by 27
  11. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  12. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  13. 2025A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworkscited by 21
  14. 2025Reason4Rec: Deliberative User Preference Alignment of Large Language Models for Recommendationcited by 21
  15. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 21
  16. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  17. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  18. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimizationcited by 17
  19. 2025Navigating the Alpha Jungle: An LLM-Powered MCTS Framework for Formulaic Factor Miningcited by 16
  20. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learningcited by 16
  21. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 16
  22. 2025Bag of Tricks for Inference-time Computation of LLM Reasoningcited by 15
  23. 2025Ontology-Guided Reverse Thinking Makes Large Language Models Stronger on Knowledge Graph Question Answeringcited by 15
  24. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  25. 2024VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifierscited by 14
  26. 2025GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Trainingcited by 14
  27. 2025Don't Get Lost in the Trees: Streamlining LLM Reasoning by Overcoming Tree Search Exploration Pitfallscited by 13
  28. 2025Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processingcited by 12
  29. 2025Better Process Supervision with Bi-directional Rewarding Signalscited by 12
  30. 2025FreePRM: Training Process Reward Models Without Ground Truth Process Labelscited by 11
  31. 2025Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoningcited by 11
  32. 2025Zero-Shot Verification-guided Chain of Thoughtscited by 10
  33. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  34. 2026DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agentcited by 10
  35. 2025Policy Guided Tree Search for Enhanced LLM Reasoningcited by 9
  36. 2024Stepwise Reasoning Error Disruption Attack of LLMscited by 8
  37. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  38. 2025SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheetscited by 8

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
  2. 2025Fast, Slow, and Tool-augmented Thinking for LLMs: A Review
  3. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
  4. 2025Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies
  5. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
  6. 2024Are Your LLMs Capable of Stable Reasoning?
  7. 2025Revisiting LLM Reasoning via Information Bottleneck
  8. 2024textttLM^texttt2: A Simple Society of Language Models Solves Complex Reasoning