mlsanityData to 7 Oct 2026

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

arXiv 2410.05229 ↗First posted 7 Oct 2024cs.LG cs.AIcited by 533

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to assess the mathematical reasoning of models on grade-school-level questions. While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics. To address these concerns, we conduct a large-scale study on several SOTA open and closed models. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models.Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and show that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is because current LLMs cannot perform genuine logical reasoning; they replicate reasoning steps from their training data. Adding a single clause that seems relevant to the question causes significant performance drops (up to 65%) across all state-of-the-art models, even though the clause doesn't contribute to the reasoning chain needed for the final answer. Overall, our work offers a more nuanced understanding of LLMs' capabilities and limitations in mathematical reasoning.

Key facts. Cited by 533 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 44 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

29 of 44 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process29 Jul
  2. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models4 Jun
  3. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic1 May
  4. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap29 Feb
  5. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers29 Feb
  6. 2024Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions17 Jan
  7. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models21 Oct

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Mistral 7Bcited by 3,065
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  10. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  11. 2022Neural Networks and the Chomsky Hierarchycited by 214
  12. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
10 more
  1. 2021Thinking Like Transformerscited by 182
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  3. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  4. 2022On the Paradox of Learning to Reason from Datacited by 93
  5. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 93
  6. 2024Can Large Language Models Reason and Plan?cited by 82
  7. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  8. 2024On Limitations of the Transformer Architecturecited by 63
  9. 2024Apple Intelligence Foundation Language Modelscited by 22
  10. 2024Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theorycited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 303; 2026: 190.

533 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2025LIMO: Less is More for Reasoningcited by 488
  3. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  4. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  5. 2025Generalizing Verifiable Instruction Followingcited by 178
  6. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  7. 2025BIG-Bench Extra Hardcited by 123
  8. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  9. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  10. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 107
  11. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
  12. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
38 more
  1. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  2. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  3. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  4. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
  5. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 64
  6. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 62
  7. 2025LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Modelscited by 61
  8. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  9. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  10. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  11. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 48
  12. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 44
  13. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  14. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lenscited by 42
  15. 2025Sleep-time Compute: Beyond Inference Scaling at Test-timecited by 41
  16. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 39
  17. 2025CRANE: Reasoning with constrained LLM generationcited by 38
  18. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  19. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  20. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physicscited by 33
  21. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 32
  22. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  23. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Modelscited by 31
  24. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  25. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  26. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 29
  27. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  28. 2025Large Language Models for Multi-Robot Systems: A Surveycited by 26
  29. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  30. 2025Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topologycited by 24
  31. 2024Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimizationcited by 23
  32. 2024Aviary: training language agents on challenging scientific taskscited by 23
  33. 2025PDE-Controller: LLMs for Autoformalization and Reasoning of PDEscited by 23
  34. 2025EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checkingcited by 23
  35. 2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidencecited by 23
  36. 2025Beyond Final Answers: Evaluating Large Language Models for Math Tutoringcited by 23
  37. 2025Large Language Models and Mathematical Reasoning Failurescited by 22
  38. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22

50 most cited of 533 All 533 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
  2. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  3. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  4. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline
  5. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges
  6. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction
  7. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  8. 2026More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning