mlsanityData to 14 Oct 2026

Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing

arXiv 2404.12253 ↗First posted 18 Apr 2024cs.CL cs.LGcited by 125

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Despite the impressive capabilities of Large Language Models (LLMs) on various tasks, they still struggle with scenarios that involves complex reasoning and planning. Recent work proposed advanced prompting techniques and the necessity of fine-tuning with high-quality data to augment LLMs' reasoning abilities. However, these approaches are inherently constrained by data availability and quality. In light of this, self-correction and self-learning emerge as viable solutions, employing strategies that allow LLMs to refine their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs in self-refining its response, particularly in complex reasoning and planning task, remains dubious. In this paper, we introduce AlphaLLM for the self-improvements of LLMs, which integrates Monte Carlo Tree Search (MCTS) with LLMs to establish a self-improving loop, thereby enhancing the capabilities of LLMs without additional annotations. Drawing inspiration from the success of AlphaGo, AlphaLLM addresses the unique challenges of combining MCTS with LLM for self-improvement, including data scarcity, the vastness search spaces of language tasks, and the subjective nature of feedback in language tasks. AlphaLLM is comprised of prompt synthesis component, an efficient MCTS approach tailored for language tasks, and a trio of critic models for precise feedback. Our experimental results in mathematical reasoning tasks demonstrate that AlphaLLM significantly enhances the performance of LLMs without additional annotations, showing the potential for self-improvement in LLMs.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 50 of its 67 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

50 of 67 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks12 Feb
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  3. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training29 Sep
  4. 2023Shepherd: A Critic for Language Model Generation8 Aug
  5. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing19 May
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  7. 2023Large Language Model Guided Tree-of-Thought15 May
  8. 2023Self-Evaluation Guided Beam Search for Reasoning1 May
  9. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  10. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
28 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  6. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  10. 2024Self-Rewarding Language Modelscited by 681
  11. 2022PAL: Program-aided Language Modelscited by 652
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  13. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  14. 2022Self-critiquing models for assisting human evaluatorscited by 380
  15. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  16. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  17. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  18. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  19. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  20. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  21. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  22. 2023Self-Alignment with Instruction Backtranslationcited by 151
  23. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 74
  24. 2017Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learningcited by 44
  25. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  26. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samplescited by 25
  27. 2024Deductive Beam Search: Decoding Deducible Rationale for Chain-of-Thought Reasoningcited by 24
  28. 2023Accelerating Monte Carlo Tree Search with Probability Tree State Abstractioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 44; 2025: 61; 2026: 20.

125 papers.

Sort:

By area:

  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  5. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  6. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  7. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  8. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  9. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 156
  10. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  11. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  12. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
38 more
  1. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  2. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 101
  3. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  4. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  5. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  6. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 64
  7. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervisioncited by 60
  8. 2025One Token to Fool LLM-as-a-Judgecited by 60
  9. 2024Large Language Model Safety: A Holistic Surveycited by 59
  10. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  11. 2024Self-evolving Agents with reflective and memory-augmented abilitiescited by 52
  12. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Searchcited by 51
  13. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  14. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  15. 2025A Survey of Large Language Model Agents for Question Answeringcited by 48
  16. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  17. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 42
  18. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  19. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 39
  20. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 37
  21. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  22. 2024HunyuanProver: A Scalable Data Synthesis Framework and Guided Tree Search for Automated Theorem Provingcited by 33
  23. 2024LiteSearch: Efficacious Tree Search for LLMcited by 31
  24. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 30
  25. 2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 30
  26. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 29
  27. 2024VideoAgent: Self-Improving Video Generationcited by 25
  28. 2024RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Modelscited by 25
  29. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  30. 2024Deliberate Reasoning in Language Models as Structure-Aware Planning with an Accurate World Modelcited by 23
  31. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehensioncited by 23
  32. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  33. 2025Solving Inequality Proofs with Large Language Modelscited by 21
  34. 2025DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversalcited by 20
  35. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  36. 2025FastMCTS: A Simple Sampling Strategy for Data Synthesiscited by 17
  37. 2024Scaling LLM Inference with Optimized Sample Compute Allocationcited by 16
  38. 2025Adaptive Self-improvement LLM Agentic System for ML Library Developmentcited by 16

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  2. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
  3. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  4. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  5. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  6. 2025Memory-Driven Self-Improvement for Decision Making with Large Language Models
  7. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
  8. 2022Large Language Models Can Self-Improve