mlsanityData to 14 Oct 2026

MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

arXiv 2309.12284 ↗First posted 21 Sep 2023cs.CL cs.AIcited by 735

On the map: Synthetic data and self-improvement for LLMs → Synthetic reasoning data and bootstrapping

Abstract (arXiv)

Large language models (LLMs) have pushed the limits of natural language understanding and exhibited excellent problem-solving ability. Despite the great success, most existing open-source LLMs (e.g., LLaMA-2) are still far away from satisfactory for solving mathematical problem due to the complex reasoning procedures. To bridge this gap, we propose MetaMath, a fine-tuned language model that specializes in mathematical reasoning. Specifically, we start by bootstrapping mathematical questions by rewriting the question from multiple perspectives without extra knowledge, which results in a new dataset called MetaMathQA. Then we fine-tune the LLaMA-2 models on MetaMathQA. Experimental results on two popular benchmarks (i.e., GSM8K and MATH) for mathematical reasoning demonstrate that MetaMath outperforms a suite of open-source LLMs by a significant margin. Our MetaMath-7B model achieves 66.4% on GSM8K and 19.4% on MATH, exceeding the state-of-the-art models of the same size by 11.5% and 8.7%. Particularly, MetaMath-70B achieves an accuracy of 82.3% on GSM8K, slightly better than GPT-3.5-Turbo. We release all the MetaMathQA dataset, the MetaMath models with different model sizes and the training code for public use.

Key facts. Cited by 735 papers in the map (data through 14 Oct 2026). Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 62 of its 84 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

62 of 84 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Large Language Models Can Self-Improve20 Oct

Other references · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  11. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,320
38 more
  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2023LIMA: Less Is More for Alignmentcited by 1,100
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  13. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  14. 2019Relational Knowledge Distillationcited by 889
  15. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  16. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  17. 2023StarCoder: may the source be with you!cited by 799
  18. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  19. 2022Galactica: A Large Language Model for Sciencecited by 744
  20. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  21. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  22. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  23. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  24. 2020Dataset Condensation with Gradient Matchingcited by 516
  25. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  26. 2021MetaICL: Learning to Learn In Contextcited by 454
  27. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  28. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  29. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  30. 2022Large Language Models Are Reasoning Teacherscited by 329
  31. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  32. 2022Teaching Small Language Models to Reasoncited by 284
  33. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  34. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  35. 2018Dataset Distillationcited by 230
  36. 2021Meta-learning via Language Model In-context Tuningcited by 164
  37. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  38. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 257; 2025: 285; 2026: 169.

735 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  4. 2025s1: Simple test-time scalingcited by 1,341
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  7. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  9. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2025LIMO: Less is More for Reasoningcited by 498
  12. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
38 more
  1. 2023Llemma: An Open Language Model For Mathematicscited by 388
  2. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  3. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  4. 2024LoRA Learns Less and Forgets Lesscited by 328
  5. 2024What matters when building vision-language models?cited by 326
  6. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  8. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  9. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  10. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  11. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  12. 2024Iterative Reasoning Preference Optimizationcited by 246
  13. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  14. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  15. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  16. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  17. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  18. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  19. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  20. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  21. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  22. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  23. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 158
  24. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  25. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
  26. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  27. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145
  28. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  29. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  30. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
  31. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  32. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  33. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  34. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  35. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  36. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 119
  37. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 119
  38. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 117

50 most cited of 735 All 735 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning
  2. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs
  3. 2025MegaMath: Pushing the Limits of Open Math Corpora
  4. 2026PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models
  5. 2025Let's Verify Math Questions Step by Step
  6. 2024PARAMANU-GANITA: Can Small Math Language Models Rival with Large Language Models on Mathematical Reasoning?
  7. 2026More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
  8. 2024Benchmarking Large Language Models for Math Reasoning Tasks