# Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models

arXiv: https://arxiv.org/abs/2507.04023 · HTML: https://mlsanity.com/paper/2507.04023/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 5 Jul 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers](https://mlsanity.com/paper/2402.19255/) (2024)
- [Phi-4-reasoning Technical Report](https://mlsanity.com/paper/2504.21318/) (2025)
- [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://mlsanity.com/paper/2503.21614/) (2025)
- [Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs](https://mlsanity.com/paper/2404.07103/) (2024)
- [Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2305.16582/) (2023)
- [Nemotron-4 15B Technical Report](https://mlsanity.com/paper/2402.16819/) (2024)
- [THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Models](https://mlsanity.com/paper/2504.13367/) (2025)
- [HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics](https://mlsanity.com/paper/2410.09988/) (2024)
- [Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models](https://mlsanity.com/paper/2505.16122/) (2025)
- [Towards Reasoning Ability of Small Language Models](https://mlsanity.com/paper/2502.11569/) (2025)
- [Let LRMs Break Free from Overthinking via Self-Braking Tuning](https://mlsanity.com/paper/2505.14604/) (2025)
- [THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models](https://mlsanity.com/paper/2505.22113/) (2025)
- [Adaptive Graph of Thoughts: Test-Time Adaptive Reasoning Unifying Chain, Tree, and Graph Structures](https://mlsanity.com/paper/2502.05078/) (2025)
- [Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models](https://mlsanity.com/paper/2502.11075/) (2025)
- [Empowering Multi-step Reasoning across Languages via Tree-of-Thoughts](https://mlsanity.com/paper/2311.08097/) (2023)
- [Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic](https://mlsanity.com/paper/2504.05262/) (2025)
- [CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models](https://mlsanity.com/paper/2504.13534/) (2025)
- [Verbosity neq Veracity: Demystify Verbosity Compensation Behavior of Large Language Models](https://mlsanity.com/paper/2411.07858/) (2024)
- [LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2506.04821/) (2025)
- [LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems](https://mlsanity.com/paper/2410.14166/) (2024)

## Built on by (most cited first, 5 of 5)

- [When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling](https://mlsanity.com/paper/2604.10739/) (2026)
- [BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models](https://mlsanity.com/paper/2509.24210/) (2025)
- [JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation](https://mlsanity.com/paper/2511.15958/) (2025)
- [Post-Reasoning: Improving the Performance of Non-Thinking Models at No Cost](https://mlsanity.com/paper/2605.06165/) (2026)
- [ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning](https://mlsanity.com/paper/2606.13316/) (2026)

## Abstract (arXiv)

Large language models (LLMs) achieve impressive performance on complex mathematical benchmarks yet sometimes fail on basic math reasoning while generating unnecessarily verbose responses. In this paper, we present LLMThinkBench, a systematic benchmark and comprehensive empirical study to evaluate the efficiency of reasoning in LLMs, focusing on the fundamental tradeoff between accuracy and overthinking. First, we formalize the accuracy-verbosity tradeoff. Second, we introduce the Overthinking Score, a harmonic-mean metric combining accuracy and token-efficiency for holistic model evaluation. Third, we establish an evaluation protocol with dynamically-generated data across 14 basic math tasks. Fourth, we conduct a large-scale empirical study evaluating 53 LLMs, including reasoning and quantized variants across different reasoning budgets. Fifth, we release LLMThinkBench as an open-source Python package and public leaderboard for reproducibility. Our findings reveal: 1) model performance on complex benchmarks does not translate directly to basic math reasoning; 2) reasoning models generate ~18x more tokens while sometimes achieving lower accuracy and exhibit catastrophic collapse when tokens are constrained, dropping by up to ~36%; 3) the accuracy-verbosity relationship is non-monotonic with extended reasoning budgets yielding diminishing returns (GPT-5/o-series models show zero accuracy gain from low -> medium -> high reasoning effort). Our findings challenge the assumption that longer reasoning in LLMs necessarily improves mathematical reasoning. Our public leaderboard is available at https://ctrl-gaurav.github.io/LLMThinkBench/. Our open-source Python package is available at https://pypi.org/project/llmthinkbench/, and the codebase can be found at https://github.com/ctrl-gaurav/LLMThinkBench for easy and reproducible evaluation.