# Large Language Models and Mathematical Reasoning Failures

arXiv: https://arxiv.org/abs/2502.11574 · HTML: https://mlsanity.com/paper/2502.11574/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers](https://mlsanity.com/paper/2402.19255/) (2024)
- [Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning](https://mlsanity.com/paper/2407.01687/) (2024)
- [U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs](https://mlsanity.com/paper/2412.03205/) (2024)

## Built on by (most cited first, 22 of 22)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Models](https://mlsanity.com/paper/2506.17114/) (2025)
- [Sherlock: Reliable and Efficient Agentic Workflow Execution](https://mlsanity.com/paper/2511.00330/) (2025)
- [Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach](https://mlsanity.com/paper/2503.10573/) (2025)
- [ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark](https://mlsanity.com/paper/2505.23851/) (2025)
- [Thinking Machines: Mathematical Reasoning in the Age of LLMs](https://mlsanity.com/paper/2508.00459/) (2025)
- [Mathematical Computation and Reasoning Errors by Large Language Models](https://mlsanity.com/paper/2508.09932/) (2025)
- [Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation](https://mlsanity.com/paper/2509.05226/) (2025)
- [Interpreting the Latent Structure of Operator Precedence in Language Models](https://mlsanity.com/paper/2510.13908/) (2025)
- [Probing Materials Knowledge in LLMs: From Latent Embeddings to Reliable Predictions](https://mlsanity.com/paper/2603.01834/) (2026)
- [Reasoning Fails Where Step Flow Breaks](https://mlsanity.com/paper/2604.06695/) (2026)
- [Even Heads Fix Odd Errors: Mechanistic Discovery and Surgical Repair in Transformer Attention](https://mlsanity.com/paper/2508.19414/) (2025)
- [Evaluating Large Language Models on Solved and Unsolved Problems in Graph Theory: Implications for Computing Education](https://mlsanity.com/paper/2602.05059/) (2026)
- [Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning](https://mlsanity.com/paper/2604.00018/) (2026)
- [SMRC: Aligning Large Language Models with Student Reasoning for Mathematical Error Correction](https://mlsanity.com/paper/2511.14684/) (2025)
- [Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis](https://mlsanity.com/paper/2602.10881/) (2026)
- [Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problem](https://mlsanity.com/paper/2603.13351/) (2026)
- [Object Search in Partially-Known Environments via LLM-informed Model-based Planning and Prompt Selection](https://mlsanity.com/paper/2603.23800/) (2026)
- [CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models](https://mlsanity.com/paper/2606.19788/) (2026)
- [Artificial Intelligence Can Match Domain Experts in Evidence Extraction and Critical Appraisal of Microbial Oncogenesis Research Publications](https://mlsanity.com/paper/2608.07250/) (2026)
- [Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification](https://mlsanity.com/paper/2609.15145/) (2026)
- [The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2610.02191/) (2026)

## Abstract (arXiv)

This paper investigates the mathematical reasoning capabilities of large language models (LLMs) using 50 newly constructed high-school-level word problems. Unlike prior studies that focus solely on answer correctness, we rigorously analyze both final answers and solution steps to identify reasoning failures. Evaluating eight state-of-the-art models - including Mixtral, Llama, Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models (e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors in spatial reasoning, strategic planning, and arithmetic, sometimes producing correct answers through flawed logic. Common failure modes include unwarranted assumptions, over-reliance on numerical patterns, and difficulty translating physical intuition into mathematical steps. Manual analysis reveals that models struggle with problems requiring multi-step deduction or real-world knowledge, despite possessing broad mathematical knowledge. Our results underscore the importance of evaluating reasoning processes, not just answers, and caution against overestimating LLMs' problem-solving proficiency. The study highlights persistent gaps in LLMs' generalization abilities, emphasizing the need for targeted improvements in structured reasoning and constraint handling.