mlsanityData to 7 Oct 2026

GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers

arXiv 2402.19255 ↗First posted 29 Feb 2024cs.CLcited by 155

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. However, there are increasing debates regarding whether these models truly understand and apply mathematical knowledge or merely rely on shortcuts for mathematical reasoning. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations. We introduce the adversarial grade school math (GSM-Plus) dataset, an extension of GSM8K augmented with various mathematical perturbations. Our experiments on 25 LLMs and 4 prompting techniques show that while LLMs exhibit different levels of math reasoning abilities, their performances are far from robust. In particular, even for problems that have been solved in GSM8K, LLMs can make mistakes when new statements are added or the question targets are altered. We also explore whether more robust performance can be achieved by composing existing prompting methods, in which we try an iterative method that generates and verifies each intermediate thought based on its reasoning goal and calculation result.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 54 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

33 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges31 Jan
  2. 2023Llemma: An Open Language Model For Mathematics16 Oct
  3. 2023Fill in the Blank: Exploring and Enhancing LLM Capabilities for Backward Reasoning in Math Word Problems3 Oct
  4. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  6. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification15 Aug
  7. 2023MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning16 Jul
  8. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks12 Apr

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
13 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  4. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  5. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  6. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  7. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  8. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  9. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  10. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  11. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  12. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
  13. 2023MathAttack: Attacking Large Language Models Towards Math Solving Abilitycited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 75; 2026: 57.

155 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  3. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  4. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  5. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  6. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  7. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  8. 2024Process Reward Model with Q-Value Rankingscited by 78
  9. 2024Self-Taught Evaluatorscited by 70
  10. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 56
  11. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  12. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
38 more
  1. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  2. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  3. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  4. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  5. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  6. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  7. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 31
  8. 2024Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoningcited by 30
  9. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  10. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 26
  11. 2025Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Modelscited by 23
  12. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 23
  13. 2024Can LLMs Solve longer Math Word Problems Better?cited by 22
  14. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Modelcited by 22
  15. 2025Large Language Models and Mathematical Reasoning Failurescited by 22
  16. 2025Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?cited by 21
  17. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  18. 2025PromptCoT: Synthesizing Olympiad-level Problems for Mathematical Reasoning in Large Language Modelscited by 20
  19. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  20. 2026Large Language Model Reasoning Failurescited by 20
  21. 2024TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Treescited by 19
  22. 2025ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoningcited by 17
  23. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 17
  24. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 15
  25. 2025Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systemscited by 14
  26. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  27. 2025ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solvingcited by 13
  28. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Modelscited by 13
  29. 2025Don't Get Lost in the Trees: Streamlining LLM Reasoning by Overcoming Tree Search Exploration Pitfallscited by 13
  30. 2026Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillationcited by 13
  31. 2024A Comprehensive Survey of Contamination Detection Methods in Large Language Modelscited by 12
  32. 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?cited by 12
  33. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  34. 2024LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvementcited by 11
  35. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  36. 2026Reward-Gated On-Policy Distillationcited by 11
  37. 2024Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passescited by 10
  38. 2025Exploring the Necessity of Reasoning in LLM-based Agent Scenarioscited by 10

50 most cited of 155 All 155 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  2. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
  3. 2024Give me a hint: Can LLMs take a hint to solve math problems?
  4. 2024Not All LLM Reasoners Are Created Equal
  5. 2023Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions
  6. 2025Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Models
  7. 2025Toward Automated Robustness Evaluation of Mathematical Reasoning
  8. 2024Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments