mlsanityData to 14 Oct 2026

AlphaMath Almost Zero: Process Supervision without Process

arXiv 2405.03553 ↗First posted 6 May 2024cs.CL cs.AIcited by 162

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Although recent advancements in large language models (LLMs) have significantly improved their performance on various tasks, they still face challenges with complex and symbolic multi-step reasoning, particularly in mathematical reasoning. To bolster the mathematical reasoning capabilities of LLMs, most existing efforts concentrate on seeking assistance from either domain experts or GPT-4 for high-quality process-supervised data, which is not only expensive but also labor-intensive. In our study, we propose an innovative framework, AlphaMath, that bypasses the need for process annotations (from humans or GPTs) by leveraging Monte Carlo Tree Search (MCTS). This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. Specifically, we integrate a value model with the LLM, automatically generating both process supervision and step-level evaluation signals in MCTS. Furthermore, we propose an efficient inference strategy, step-level beam search, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective reasoning paths, rather than solely relying on prior probabilities. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous state-of-the-art methods.

Key facts. Cited by 162 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 51 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

34 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Let's Verify Step by Step31 May

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
21 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  6. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  7. 2022PAL: Program-aided Language Modelscited by 652
  8. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  9. 2023Llemma: An Open Language Model For Mathematicscited by 388
  10. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  11. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  12. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  13. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  14. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  15. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  16. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 77
  17. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  18. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Datacited by 43
  19. 2024MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkitcited by 14
  20. 2024Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibrationcited by 13
  21. 2024SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 34; 2025: 109; 2026: 19.

162 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  2. 2025LIMO: Less is More for Reasoningcited by 498
  3. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  5. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  6. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  7. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  8. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  9. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  10. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  11. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  12. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
38 more
  1. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 158
  2. 2024Tree Search for Language Model Agentscited by 151
  3. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  4. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  5. 2025Process Reward Models That Thinkcited by 118
  6. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  7. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  8. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
  9. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  10. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  11. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  12. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  13. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  14. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  15. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  16. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 70
  17. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 62
  18. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 61
  19. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 59
  20. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  21. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 54
  22. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  23. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Searchcited by 51
  24. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  25. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 50
  26. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 47
  27. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 47
  28. 2025Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?cited by 44
  29. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  30. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 44
  31. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  32. 2024Entropy-Regularized Process Reward Modelcited by 38
  33. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 38
  34. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 37
  35. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 34
  36. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 31
  37. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 29
  38. 2024Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Criticcited by 28

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision
  2. 2025From Static to Dynamic: Adaptive Monte Carlo Search for Mathematical Process Supervision
  3. 2025Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
  4. 2026Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain
  5. 2026Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning
  6. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
  7. 2025Improving Value-based Process Verifier via Low-Cost Variance Reduction
  8. 2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling