mlsanityData to 7 Oct 2026

Not All LLM Reasoners Are Created Equal

arXiv 2410.01748 ↗First posted 2 Oct 2024cs.LGcited by 29

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We study the depth of grade-school math (GSM) problem-solving capabilities of LLMs. To this end, we evaluate their performance on pairs of existing math word problems together so that the answer to the second problem depends on correctly answering the first problem. Our findings reveal a significant reasoning gap in most LLMs, that is performance difference between solving the compositional pairs and solving each question independently. This gap is more pronounced in smaller, more cost-efficient, and math-specialized models. Moreover, instruction-tuning recipes and code generation have varying effects across LLM sizes, while finetuning on GSM can lead to task overfitting. Our analysis indicates that large reasoning gaps are not because of test-set leakage, but due to distraction from additional context and poor second-hop reasoning. Overall, LLMs exhibit systematic differences in their reasoning abilities, despite what their performance on standard benchmarks indicates.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 62 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

38 of 62 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024AI-Assisted Generation of Difficult Math Questions30 Jul
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic1 May
  3. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap29 Feb
  4. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges31 Jan
  5. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models3 Aug

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2024Mixtral of Expertscited by 1,785
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  12. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
20 more
  1. 2021Scaling Laws for Transfercited by 265
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  3. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  4. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  5. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  6. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  7. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  8. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  9. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  10. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  11. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  12. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  13. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Modelscited by 38
  14. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
  15. 2024Break the Chain: Large Language Models Can be Shortcut Reasonerscited by 32
  16. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  17. 2024Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoningcited by 17
  18. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14
  19. 2024Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?cited by 10
  20. 2021Reasoning Chain Based Adversarial Attack for Multi-hop Question Answeringcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 9.

29 papers.

Sort:

By area:

  1. 2025Humanity's Last Examcited by 467
  2. 2025BIG-Bench Extra Hardcited by 123
  3. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  4. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  5. 2026Large Language Model Reasoning Failurescited by 20
  6. 2025ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoningcited by 17
  7. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  8. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 15
  9. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  10. 2024Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspectivecited by 9
  11. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMscited by 9
  12. 2025Scaling can lead to compositional generalizationcited by 7
17 more
  1. 2025Hell or High Water: Evaluating Agentic Recovery from External Failurescited by 6
  2. 2025None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarkscited by 5
  3. 2025Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Modelscited by 5
  4. 2025Multi-Step Reasoning in Korean and the Emergent Miragecited by 4
  5. 2025RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflowscited by 3
  6. 2025Peeking Behind Closed Doors: Risks of LLM Evaluation by Private Data Curatorscited by 2
  7. 2025Can Memory-Augmented Language Models Generalize on Reasoning-in-a-Haystack Tasks?cited by 2
  8. 2025RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrievalcited by 1
  9. 2025Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoningcited by 1
  10. 2026How Does Unfaithful Reasoning Emerge from Autoregressive Training? A Study of Synthetic Experimentscited by 1
  11. 2026LEAD: Breaking the No-Recovery Bottleneck in Long-Horizon Reasoningcited by 1
  12. 2026How Psychological Learning Paradigms Shaped and Constrained Artificial Intelligencecited by 0
  13. 2026Agentic Frameworks for Reasoning Tasks: An Empirical Studycited by 0
  14. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  15. 2026WordPolo: Evaluating Language Models Through Iterative Semantic Feedbackcited by 0
  16. 2026Differentiable Fuzzy Inference Layer: A Monotone, Compositional Ordinal Reasoning Head for Large Language Modelscited by 0
  17. 2026Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
  2. 2024An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning
  3. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
  4. 2024Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning
  5. 2023Fill in the Blank: Exploring and Enhancing LLM Capabilities for Backward Reasoning in Math Word Problems
  6. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities
  7. 2026A Four-Stage Decomposition of Word-Problem Solving and Mechanistic Fragility in LLM Math Reasoning
  8. 2024LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems