mlsanityData to 14 Oct 2026

The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic

arXiv 2605.28700 ↗First posted 27 May 2026cs.AI cs.CLcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

The GSM-Symbolic benchmark (Mirzadeh et al., 2025) reported consistent performance drops across 25 Large Language Models (LLMs) when tested on template-generated variants of GSM8K problems, concluding that the models lack genuine reasoning capabilities. We argue that this conclusion rests on shaky statistical ground. Re-evaluating 20 open-weight models using bootstrapped Generalised Linear Mixed Models with per-question random effects, we find that only 8 exhibit statistically significant performance changes under the original prompt format. Moreover, we identify a previously unacknowledged factor: the distribution of integers in problem texts of the main GSM-Symbolic dataset is systematically shifted towards larger values relative to the original GSM8K (K-S statistic = 0.12, p < 0.001), contradicting the original authors' claims. Controlling for this large-number effect accounts for significance in half of the remaining cases. Among models with statistically significant performance deltas, we identify distinct, model-specific behavioural failure profiles -- including fragility of variable binding, arithmetic limitations, and dual-task interference -- underscoring that blanket claims about LLM reasoning risk being both statistically premature and mechanistically misleading.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 63 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

21 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges12 Feb
  2. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models7 Oct
  3. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers29 Feb

Other references · 18

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  8. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  9. 2020With Little Power Comes Great Responsibilitycited by 142
  10. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  11. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  12. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
6 more
  1. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  2. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 22
  3. 2025Statistical Multicriteria Evaluation of LLM-Generated Textcited by 9
  4. 2024A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutionscited by 8
  5. 2025Efficient numeracy in language models through single-token number embeddingscited by 7
  6. 2026Did You Forget What I Asked? Prospective Memory Failures in Large Language Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?
  2. 2026GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentations
  3. 2024Investigating Symbolic Capabilities of Large Language Models
  4. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  5. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
  6. 2024Can Large Language Models Act as Symbolic Reasoners?
  7. 2025Evaluating LLM Reasoning Beyond Correctness and CoT
  8. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic