mlsanityData to 7 Oct 2026

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

arXiv 2501.04519 ↗First posted 8 Jan 2025cs.CLcited by 308

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

We present rStar-Math to demonstrate that small language models (SLMs) can rival or even surpass the math reasoning capability of OpenAI o1, without distillation from superior models. rStar-Math achieves this by exercising "deep thinking" through Monte Carlo Tree Search (MCTS), where a math policy SLM performs test-time search guided by an SLM-based process reward model. rStar-Math introduces three innovations to tackle the challenges in training the two SLMs: (1) a novel code-augmented CoT data sythesis method, which performs extensive MCTS rollouts to generate step-by-step verified reasoning trajectories used to train the policy SLM; (2) a novel process reward model training method that avoids na"ive step-level score annotation, yielding a more effective process preference model (PPM); (3) a self-evolution recipe in which the policy SLM and PPM are built from scratch and iteratively evolved to improve reasoning capabilities. Through 4 rounds of self-evolution with millions of synthesized solutions for 747k math problems, rStar-Math boosts SLMs' math reasoning to state-of-the-art levels. On the MATH benchmark, it improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad (AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among the top 20% the brightest high school math students. Code and data will be available at https://github.com/microsoft/rStar.

Key facts. Cited by 308 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 51 references that are in the map. First posted on arXiv 8 Jan 2025.

Builds on

39 of 51 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data2 Oct
  2. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B11 Jun
  3. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time25 May
  4. 2024Common 7B Language Models Already Possess Strong Math Capabilities7 Mar
  5. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoning5 Mar
  6. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline16 Jan
  7. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning5 Oct
  8. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  10. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models3 Aug

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
17 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  3. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  4. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  5. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  6. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  7. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  8. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  9. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  10. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  11. 2024DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Datacited by 225
  12. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  13. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  14. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  15. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 90
  16. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85
  17. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 223; 2026: 85.

308 papers.

Sort:

By area:

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  4. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  5. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  7. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  8. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  9. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 150
  10. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  11. 2025Phi-4-reasoning Technical Reportcited by 146
  12. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
38 more
  1. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  2. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  3. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 118
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  5. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
  6. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 97
  7. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Modelscited by 93
  8. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tcited by 72
  9. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  10. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  11. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  12. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
  13. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 60
  14. 2025TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoningcited by 59
  15. 2025ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoningcited by 57
  16. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  17. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  18. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53
  19. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  20. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
  21. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 49
  22. 2025Chain-of-Retrieval Augmented Generationcited by 48
  23. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Searchcited by 48
  24. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  25. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 48
  26. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  27. 2025Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modelingcited by 44
  28. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  29. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  30. 2025Reasoning Language Models: A Blueprintcited by 41
  31. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 39
  32. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  33. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 39
  34. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  35. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  36. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learningcited by 37
  37. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
  38. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36

50 most cited of 308 All 308 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math
  2. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Data
  3. 2024MC-NEST: Enhancing Mathematical Reasoning in Large Language Models leveraging a Monte Carlo Self-Refine Tree
  4. 2025Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving
  5. 2023Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning
  6. 2025Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets
  7. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
  8. 2024Small Language Models are Equation Reasoners