mlsanityData to 7 Oct 2026

Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning

arXiv 2405.00451 ↗First posted 1 May 2024cs.AI cs.LGcited by 238

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process inspired by the successful strategy employed by AlphaZero. Our work leverages Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. To enhance consistency in intermediate steps, we combine outcome validation and stepwise self-evaluation, continually updating the quality assessment of newly generated data. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data. Theoretical analysis reveals the importance of using on-policy sampled data for successful self-improving. Extensive evaluations on various arithmetic and commonsense reasoning tasks demonstrate remarkable performance improvements over existing models. For instance, our approach outperforms the Mistral-7B Supervised Fine-Tuning (SFT) baseline on GSM8K, MATH, and ARC-C, with substantial increases in accuracy to 81.8% (+5.9%), 34.7% (+5.8%), and 76.4% (+15.8%), respectively. Additionally, our research delves into the training and inference compute tradeoff, providing insights into how our method effectively maximizes performance gains. Our code is publicly available at https://github.com/YuxiXie/MCTS-DPO.

Key facts. Cited by 238 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 60 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

46 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Models28 Oct

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
33 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  13. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  14. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  15. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  16. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  17. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  18. 2022Large Language Models Can Self-Improvecited by 656
  19. 2024Self-Rewarding Language Modelscited by 654
  20. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  21. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  22. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  23. 2020Rethinking Pre-training and Self-trainingcited by 352
  24. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  25. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  26. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  27. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  28. 2022Formal Mathematics Statement Curriculum Learningcited by 141
  29. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  30. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 74
  31. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 59
  32. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  33. 2023Crystal: Introspective Reasoners Reinforced with Self-Feedbackcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 152; 2026: 49.

238 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  4. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  5. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  6. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  7. 2024Token-Budget-Aware LLM Reasoningcited by 242
  8. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  9. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  10. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  11. 2024Tree Search for Language Model Agentscited by 148
  12. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
38 more
  1. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  2. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  3. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  4. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  5. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  6. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  7. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  8. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  9. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  10. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  11. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  12. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 75
  13. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  14. 2025Self-rewarding correction for mathematical reasoningcited by 67
  15. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  16. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 66
  17. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 62
  18. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  19. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  20. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  21. 2024Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideascited by 52
  22. 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learningcited by 51
  23. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  24. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  25. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 43
  26. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  27. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  28. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  29. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  30. 2025Video-T1: Test-Time Scaling for Video Generationcited by 39
  31. 2024Entropy-Regularized Process Reward Modelcited by 38
  32. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  33. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
  34. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  35. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  36. 2025A Survey of Direct Preference Optimizationcited by 35
  37. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  38. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Explorationcited by 33

50 most cited of 238 All 238 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Advancing Process Verification for Large Language Models via Tree-Based Preference Learning
  2. 2025SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition
  3. 2024Enhancing LLM Reasoning with Reward-guided Tree Search
  4. 2024MC-NEST: Enhancing Mathematical Reasoning in Large Language Models leveraging a Monte Carlo Self-Refine Tree
  5. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Function
  6. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
  7. 2024Step-level Value Preference Optimization for Mathematical Reasoning
  8. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning