mlsanityData to 7 Oct 2026

MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

arXiv 2309.05653 ↗First posted 11 Sep 2023cs.CLcited by 483

Not on the map. Why?

Cited by 237 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce MAmmoTH, a series of open-source large language models (LLMs) specifically tailored for general math problem-solving. The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset. MathInstruct is compiled from 13 math datasets with intermediate rationales, six of which have rationales newly curated by us. It presents a unique hybrid of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT not only unleashes the potential of tool use but also allows different thought processes for different math problems. As a result, the MAmmoTH series substantially outperform existing open-source models on nine mathematical reasoning datasets across all scales with an average accuracy gain between 16% and 32%. Remarkably, our MAmmoTH-7B model reaches 33% on MATH (a competition-level dataset), which exceeds the best open-source 7B model (WizardMath) by 23%, and the MAmmoTH-34B model achieves 44% accuracy on MATH, even surpassing GPT-4's CoT result. Our work underscores the importance of diverse problem coverage and the use of hybrid rationales in developing superior math generalist models.

Key facts. Cited by 483 papers in the map (data through 7 Oct 2026). Builds on 64 of its 76 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

64 of 76 references.

References · 64

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  15. 2023LIMA: Less Is More for Alignmentcited by 1,080
  16. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  17. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  18. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  19. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  20. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  21. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  22. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  23. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  24. 2022Galactica: A Large Language Model for Sciencecited by 736
  25. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  26. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  27. 2023Instruction Tuning with GPT-4cited by 676
  28. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  29. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  30. 2022PAL: Program-aided Language Modelscited by 635
  31. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  32. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  33. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  34. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  35. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  36. 2016Solving General Arithmetic Word Problemscited by 353
  37. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  38. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 240; 2025: 155; 2026: 54.

483 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  8. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  9. 2025LIMO: Less is More for Reasoningcited by 488
  10. 2025Process Reinforcement through Implicit Rewardscited by 478
  11. 2023Llemma: An Open Language Model For Mathematicscited by 385
  12. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
38 more
  1. 2025Kimi-VL Technical Reportcited by 381
  2. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  4. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  5. 2024What matters when building vision-language models?cited by 321
  6. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  7. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  8. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  10. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  11. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  12. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  13. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  14. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  15. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  16. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  17. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  18. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  19. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  20. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  21. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  22. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  23. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  24. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  25. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 155
  26. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  27. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  28. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  29. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  30. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  31. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  32. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  33. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  34. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  35. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  36. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  37. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  38. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130

50 most cited of 483 All 483 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluation of LLMs for mathematical problem solving
  2. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data
  3. 2024PARAMANU-GANITA: Can Small Math Language Models Rival with Large Language Models on Mathematical Reasoning?
  4. 2024MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning
  5. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  6. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning
  7. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
  8. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time