mlsanityData to 7 Oct 2026

Common 7B Language Models Already Possess Strong Math Capabilities

arXiv 2403.04706 ↗First posted 7 Mar 2024cs.CL cs.AIcited by 108

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Mathematical capabilities were previously believed to emerge in common language models only at a very large scale or require extensive math-related pre-training. This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities, as evidenced by its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks, respectively, when selecting the best response from 256 random generations. The primary issue with the current base model is the difficulty in consistently eliciting its inherent mathematical capabilities. Notably, the accuracy for the first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks, respectively. We find that simply scaling up the SFT data can significantly enhance the reliability of generating correct answers. However, the potential for extensive scaling is constrained by the scarcity of publicly available math questions. To overcome this limitation, we employ synthetic data, which proves to be nearly as effective as real data and shows no clear saturation when scaled up to approximately one million samples. This straightforward approach achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B models, surpassing previous models by 14.2% and 20.8%, respectively. We also provide insights into scaling behaviors across different reasoning complexities and error types.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 30 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

21 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  2. 2023Llemma: An Open Language Model For Mathematics16 Oct
  3. 2023MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning9 Oct
  4. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug

Other references · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
4 more
  1. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  2. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  3. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  4. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 52; 2025: 46; 2026: 10.

108 papers.

Sort:

By area:

  1. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  2. 2025LIMO: Less is More for Reasoningcited by 488
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  6. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  7. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  8. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  9. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  10. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  11. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  12. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
38 more
  1. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  2. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  3. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  4. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  5. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  6. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  7. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  8. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  9. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  10. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
  11. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  12. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  13. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  14. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  15. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  16. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  17. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  18. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  19. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  20. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 39
  21. 2024AI-Assisted Generation of Difficult Math Questionscited by 37
  22. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 32
  23. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  24. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  25. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  26. 2024Reverse Thinking Makes LLMs Stronger Reasonerscited by 29
  27. 2025Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspectivecited by 29
  28. 2025Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Propertiescited by 28
  29. 2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 27
  30. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  31. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  32. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  33. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  34. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  35. 2024U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMscited by 24
  36. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehensioncited by 23
  37. 2024MuMath-Code: Combining Tool-Use Large Language Models with Multi-perspective Data Augmentation for Mathematical Reasoningcited by 22
  38. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 21

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  3. 2025Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
  4. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
  5. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data
  6. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
  7. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline
  8. 2024Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange