mlsanityData to 7 Oct 2026

MAmmoTH2: Scaling Instructions from the Web

arXiv 2405.03548 ↗First posted 6 May 2024cs.CLcited by 143

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on this dataset, we build MAmmoTH2 models, which significantly boost performance on reasoning benchmarks. Notably, MAmmoTH2-7B's (Mistral) performance increases from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training on any in-domain data. Further training MAmmoTH2 on public instruction tuning datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing a new paradigm for building better instruction tuning data.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 58 of its 70 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

58 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Advancing LLM Reasoning Generalists with Preference Trees2 Apr
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  3. 2022PAL: Program-aided Language Models18 Nov
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023Mistral 7Bcited by 3,065
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2024Mixtral of Expertscited by 1,785
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  13. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  14. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  15. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  16. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  17. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  18. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  19. 2022Galactica: A Large Language Model for Sciencecited by 736
  20. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  21. 2024Yi: Open Foundation Models by 01.AIcited by 706
  22. 2023Instruction Tuning with GPT-4cited by 676
  23. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  24. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  25. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  26. 2023Llemma: An Open Language Model For Mathematicscited by 385
  27. 2016FastText.zip: Compressing text classification modelscited by 331
  28. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  29. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  30. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  31. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  32. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  33. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  34. 2023Self-Alignment with Instruction Backtranslationcited by 149
  35. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  36. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  37. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  38. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 73; 2026: 17.

143 papers.

Sort:

By area:

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  3. 2025LIMO: Less is More for Reasoningcited by 488
  4. 2025Process Reinforcement through Implicit Rewardscited by 478
  5. 2024LoRA Learns Less and Forgets Lesscited by 313
  6. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  7. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  8. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  9. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150
  10. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  11. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  12. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
38 more
  1. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  2. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  3. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  4. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  5. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  6. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  7. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  8. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  9. 2025Reinforcing General Reasoning without Verifierscited by 76
  10. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  11. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  12. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  13. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  14. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
  15. 2025One Token to Fool LLM-as-a-Judgecited by 56
  16. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  17. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  18. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Predictioncited by 54
  19. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  20. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  21. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  22. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 47
  23. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  24. 2025TaskCraft: Automated Generation of Agentic Taskscited by 46
  25. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 46
  26. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  27. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  28. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  29. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  30. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  31. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  32. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 37
  33. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  34. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  35. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
  36. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  37. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 32
  38. 2024Evaluating Language Models as Synthetic Data Generatorscited by 32

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Instruction Tuning with Pre-training
  2. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  3. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  4. 2026DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning
  5. 2025Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
  6. 2025Big Reasoning with Small Models: Instruction Retrieval at Inference Time
  7. 2024Dual Instruction Tuning with Large Language Models for Mathematical Reasoning
  8. 2023Tuna: Instruction Tuning using Feedback from Large Language Models