mlsanityData to 14 Oct 2026

MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations

arXiv 2502.06453 ↗First posted 10 Feb 2025cs.LG cs.AI cs.CLcited by 89

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models have demonstrated impressive performance on challenging mathematical reasoning tasks, which has triggered the discussion of whether the performance is achieved by true reasoning capability or memorization. To investigate this question, prior work has constructed mathematical benchmarks when questions undergo simple perturbations -- modifications that still preserve the underlying reasoning patterns of the solutions. However, no work has explored hard perturbations, which fundamentally change the nature of the problem so that the original solution steps do not apply. To bridge the gap, we construct MATH-P-Simple and MATH-P-Hard via simple perturbation and hard perturbation, respectively. Each consists of 279 perturbed math problems derived from level-5 (hardest) problems in the MATH dataset (Hendrycksmath et. al., 2021). We observe significant performance drops on MATH-P-Hard across various models, including o1-mini (-16.49%) and gemini-2.0-flash-thinking (-12.9%). We also raise concerns about a novel form of memorization where models blindly apply learned problem-solving skills without assessing their applicability to modified contexts. This issue is amplified when using original problems for in-context learning. We call for research efforts to address this challenge, which is critical for developing more robust and reliable reasoning models.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 59 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

48 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics28 Oct
  2. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models7 Oct
  3. 2024AI-Assisted Generation of Difficult Math Questions30 Jul
  4. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist11 Jul
  5. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic1 May
  6. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap29 Feb
  7. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers29 Feb
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb

Other references · 40

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2020Extracting Training Data from Large Language Modelscited by 2,189
28 more
  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  8. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  12. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  13. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  14. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  15. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  16. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  17. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  18. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  19. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  20. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  21. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
  22. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  23. 2022The Privacy Onion Effect: Memorization is Relativecited by 136
  24. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  25. 2023Copyright Violations and Large Language Modelscited by 108
  26. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 43
  27. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
  28. 2023Privacy Implications of Retrieval-Based Language Modelscited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 51; 2026: 38.

89 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  4. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  5. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 33
  6. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  7. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  8. 2025Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Modelscited by 23
  9. 2026Large Language Model Reasoning Failurescited by 21
  10. 2025Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Modelscited by 20
  11. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  12. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
38 more
  1. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  2. 2025Patterns Over Principles: The Fragility of Inductive Reasoning in LLMs under Noisy Observationscited by 12
  3. 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solvingcited by 12
  4. 2025Can LLMs Explain Themselves Counterfactually?cited by 10
  5. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 10
  6. 2025Can Theoretical Physics Research Benefit from Language Agents?cited by 9
  7. 2025An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problemscited by 8
  8. 2025Debunk the Myth of SFT Generalizationcited by 8
  9. 2025Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learningcited by 7
  10. 2025None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarkscited by 6
  11. 2025Benford's Curse: Tracing Digit Bias to Numerical Hallucination in LLMscited by 6
  12. 2025Toward Automated Robustness Evaluation of Mathematical Reasoningcited by 6
  13. 2025Skill-Targeted Adaptive Trainingcited by 6
  14. 2025How Well Do Large Reasoning Models Translate? A Comprehensive Evaluation for Multi-Domain Machine Translationcited by 5
  15. 2025ASyMOB: Algebraic Symbolic Mathematical Operations Benchmarkcited by 5
  16. 2025Decoupling Knowledge and Reasoning in LLMs: An Exploration Using Cognitive Dual-System Theorycited by 5
  17. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 5
  18. 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scalingcited by 5
  19. 2026CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVRcited by 5
  20. 2025Executable Functional Abstractions: Inferring Generative Programs for Advanced Math Problemscited by 4
  21. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  22. 2025CritiCal: Can Critique Help LLM Uncertainty or Confidence Calibration?cited by 4
  23. 2025The Ramon Llull's Thinking Machine for Automated Ideationcited by 3
  24. 2025Test of Time: Rethinking Temporal Signal of Benchmark Contaminationcited by 3
  25. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  26. 2025Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Modelscited by 3
  27. 2025ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challengescited by 3
  28. 2026Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problemscited by 3
  29. 2026Robust Reasoning Benchmarkcited by 3
  30. 2025PromptPrism: A Linguistically-Inspired Taxonomy for Promptscited by 2
  31. 2025Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understandingcited by 2
  32. 2025ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluationcited by 2
  33. 2026Finding the Cracks: Improving LLMs Reasoning with Paraphrastic Probing and Consistency Verificationcited by 2
  34. 2026Code2Math: Can Your Code Agent Effectively Evolve Math Problems Through Exploration?cited by 2
  35. 2026GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Modelscited by 2
  36. 2026Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarkscited by 2
  37. 2025Teaching Small Language Models to Learn Logic through Meta-Learningcited by 1
  38. 2025Robustness of Neurosymbolic Reasoners on First-Order Logic Problemscited by 1

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities
  2. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  3. 2023Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions
  4. 2025Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
  5. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  6. 2025Can Large Reasoning Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking?
  7. 2025Exploring LLM Reasoning Through Controlled Prompt Variations
  8. 2025Examining False Positives under Inference Scaling for Mathematical Reasoning