mlsanityData to 7 Oct 2026

MathScale: Scaling Instruction Tuning for Mathematical Reasoning

arXiv 2403.02884 ↗First posted 5 Mar 2024cs.CL cs.AI cs.LGcited by 144

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities in problem-solving. However, their proficiency in solving mathematical problems remains inadequate. We propose MathScale, a simple and scalable method to create high-quality mathematical reasoning data using frontier LLMs (e.g., tt GPT-3.5). Inspired by the cognitive mechanism in human mathematical learning, it first extracts topics and knowledge points from seed math questions and then build a concept graph, which is subsequently used to generate new math questions. MathScale exhibits effective scalability along the size axis of the math dataset that we generate. As a result, we create a mathematical reasoning dataset (MathScaleQA) containing two million math question-answer pairs. To evaluate mathematical reasoning abilities of LLMs comprehensively, we construct sc MwpBench, a benchmark of Math Word Problems, which is a collection of ten datasets (including GSM8K and MATH) covering K-12, college, and competition level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g., LLaMA-2 and Mistral), resulting in significantly improved capabilities in mathematical reasoning. Evaluated on sc MwpBench, MathScale-7B achieves state-of-the-art performance across all datasets, surpassing its best peers of equivalent size by 42.9% in micro average accuracy and 43.7% in macro average accuracy, respectively.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 30 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

14 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug

Other references · 12

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023Mistral 7Bcited by 3,065
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2016Sequence-Level Knowledge Distillationcited by 761
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  9. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  11. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  12. 2020Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problemscited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 78; 2026: 34.

144 papers.

Sort:

By area:

  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  3. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  4. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  5. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  6. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  7. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  8. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  9. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  10. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 87
  11. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  12. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 53
38 more
  1. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  2. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  3. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  4. 2024LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languagescited by 46
  5. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 45
  6. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  7. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  8. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  9. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
  10. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 39
  11. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  12. 2025R-PRM: Reasoning-Driven Process Reward Modelingcited by 38
  13. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 37
  14. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  15. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  16. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 32
  17. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  18. 2025BitNet b1.58 2B4T Technical Reportcited by 30
  19. 2024MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactionscited by 26
  20. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 26
  21. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoningcited by 26
  22. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 25
  23. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  24. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  25. 2025Learning to Reason at the Frontier of Learnabilitycited by 20
  26. 2025PromptCoT: Synthesizing Olympiad-level Problems for Mathematical Reasoning in Large Language Modelscited by 20
  27. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 20
  28. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuningcited by 20
  29. 2024TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Treescited by 19
  30. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  31. 2024DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Searchcited by 17
  32. 2025FastMCTS: A Simple Sampling Strategy for Data Synthesiscited by 16
  33. 2024Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoningcited by 15
  34. 2024LogicPro: Improving Complex Logical Reasoning via Program-Guided Learningcited by 14
  35. 2024Exploring the Mystery of Influential Data for Mathematical Reasoningcited by 13
  36. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Modelscited by 13
  37. 2025Theorem Prover as a Judge for Synthetic Data Generationcited by 13
  38. 2025GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMscited by 13

50 most cited of 144 All 144 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  2. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities
  3. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes On
  4. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges
  5. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
  6. 2023Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions
  7. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning
  8. 2024LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning