mlsanityData to 7 Oct 2026

A Careful Examination of Large Language Model Performance on Grade School Arithmetic

arXiv 2405.00332 ↗First posted 1 May 2024cs.CL cs.AI cs.LGcited by 188

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes. Further analysis suggests a positive relationship (Spearman's r^2 = 0.36) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that some models may have partially memorized GSM8k. Nevertheless, many models, especially those on the frontier, show minimal signs of overfitting, and all models broadly demonstrate generalization to novel math problems guaranteed to not be in their training data.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 76 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

24 of 76 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap29 Feb
  2. 2023Llemma: An Open Language Model For Mathematics16 Oct

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
10 more
  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2022PAL: Program-aided Language Modelscited by 635
  4. 2023Textbooks Are All You Needcited by 561
  5. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  6. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  7. 2022Data Contamination: From Memorization to Exploitationcited by 234
  8. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  9. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  10. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 57; 2025: 75; 2026: 56.

188 papers.

Sort:

By area:

  1. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  2. 2025LIMO: Less is More for Reasoningcited by 488
  3. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  4. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  5. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  6. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  7. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  8. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  9. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  10. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  11. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  12. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
38 more
  1. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  2. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  3. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  4. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  5. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  6. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  7. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  8. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  9. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  10. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  11. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 45
  12. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 42
  13. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  14. 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
  15. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
  16. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 41
  17. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
  18. 2024AI-Assisted Generation of Difficult Math Questionscited by 37
  19. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
  20. 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
  21. 2025A Survey on Data Contamination for Large Language Modelscited by 33
  22. 2024Not All LLM Reasoners Are Created Equalcited by 29
  23. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  24. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languagescited by 28
  25. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 28
  26. 2025The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofscited by 27
  27. 2024RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Contentcited by 26
  28. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 26
  29. 2024What Are the Odds? Language Models Are Capable of Probabilistic Reasoningcited by 25
  30. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 25
  31. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  32. 2025Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topologycited by 24
  33. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  34. 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 23
  35. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
  36. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  37. 2025Adaptive LLM Routing under Budget Constraintscited by 21
  38. 2025PromptCoT: Synthesizing Olympiad-level Problems for Mathematical Reasoning in Large Language Modelscited by 20

50 most cited of 188 All 188 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
  2. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  3. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitions
  4. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  5. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
  6. 2025MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning
  7. 2025Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
  8. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction