mlsanityData to 7 Oct 2026

TinyGSM: achieving >80% on GSM8k with small language models

arXiv 2312.09241 ↗First posted 14 Dec 2023cs.LG cs.CLcited by 68

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Small-scale models offer various computational advantages, and yet to which extent size is critical for problem-solving abilities remains an open question. Specifically for solving grade school math, the smallest model size so far required to break the 80% barrier on the GSM8K benchmark remains to be 34B. Our work studies how high-quality datasets may be the key for small language models to acquire mathematical reasoning. We introduce textttTinyGSM, a synthetic dataset of 12.3M grade school math problems paired with Python solutions, generated fully by GPT-3.5. After finetuning on textttTinyGSM, we find that a duo of a 1.3B generation model and a 1.3B verifier model can achieve 81.5% accuracy, outperforming existing models that are orders of magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher'' model (77.4%), from which our model's training data is generated. Our approach is simple and has two key components: 1) the high-quality dataset textttTinyGSM, 2) the use of a verifier, which selects the final outputs from multiple candidate generations.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 48 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

43 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning16 Nov
  2. 2023Llemma: An Open Language Model For Mathematics16 Oct
  3. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  5. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification15 Aug
  6. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions28 May

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
25 more
  1. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  3. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  5. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  6. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  7. 2023Textbooks Are All You Needcited by 561
  8. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  9. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  10. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  11. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  12. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  13. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  14. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  15. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  16. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  17. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  18. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  19. 2023The False Promise of Imitating Proprietary LLMscited by 254
  20. 2023Self-Consuming Generative Models Go MADcited by 239
  21. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  22. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  23. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  24. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  25. 2023Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luckcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 11; 2026: 15.

68 papers.

Sort:

By area:

  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  4. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  5. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  6. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  7. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  8. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  9. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  10. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  11. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  12. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
38 more
  1. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  2. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 65
  3. 2024Augmenting Math Word Problems via Iterative Question Composingcited by 58
  4. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  5. 2024TinyAgent: Function Calling at the Edgecited by 55
  6. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  7. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 51
  8. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  9. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 49
  10. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 49
  11. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  12. 2025Reasoning to Learn from Latent Thoughtscited by 43
  13. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
  14. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  15. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  16. 2024Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problemscited by 31
  17. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  18. 2024Automatic Instruction Evolving for Large Language Modelscited by 24
  19. 2024Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Modelscited by 20
  20. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
  21. 2024Proposing and solving olympiad geometry with guided tree searchcited by 18
  22. 2024Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Trainingcited by 17
  23. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  24. 2024MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generationcited by 13
  25. 2024Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoningcited by 13
  26. 2024Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Modelscited by 12
  27. 2024Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generationcited by 11
  28. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMscited by 10
  29. 2024Merging Improves Self-Critique Against Jailbreak Attackscited by 8
  30. 2026Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Trainingcited by 8
  31. 2026Posterior Refinement: Fast Language Generation via Any-Order Flow Mapscited by 8
  32. 2024System-2 Mathematical Reasoning via Enriched Instruction Tuningcited by 7
  33. 2024LEMoN: Label Error Detection using Multimodal Neighborscited by 6
  34. 2024SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Modelscited by 6
  35. 2024Small Language Models for Application Interactions: A Case Studycited by 5
  36. 2024Mathfish: Evaluating Language Model Math Reasoning via Grounding in Educational Curriculacited by 5
  37. 2024SoftSRV: Learn to Generate Targeted Synthetic Datacited by 3
  38. 2026Self-Supervised Bootstrapping of Action-Predictive Embodied Reasoningcited by 3

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  2. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
  3. 2025Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
  4. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline
  5. 2026Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models
  6. 2024Common 7B Language Models Already Possess Strong Math Capabilities
  7. 2025Evaluation of LLMs for mathematical problem solving
  8. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges