mlsanityData to 14 Oct 2026

OptimalThinkingBench: Evaluating Over and Underthinking in LLMs

arXiv 2508.13141 ↗First posted 18 Aug 2025cs.CL cs.LGcited by 35

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Thinking LLMs solve complex tasks at the expense of increased compute and overthinking on simpler problems, while non-thinking LLMs are faster and cheaper but underthink on harder reasoning problems. This has led to the development of separate thinking and non-thinking LLM variants, leaving the onus of selecting the optimal model for each query on the end user. We introduce OptimalThinkingBench, a unified benchmark that jointly evaluates overthinking and underthinking in LLMs and also encourages the development of optimally-thinking models that balance performance and efficiency. Our benchmark comprises two sub-benchmarks: OverthinkingBench, featuring simple math and general queries in 72 domains, and UnderthinkingBench, containing 11 challenging reasoning tasks along with harder math problems. Using novel thinking-adjusted accuracy metrics, we extensively evaluate 33 different thinking and non-thinking models and show that no model is able to optimally think on our benchmark. Thinking models often overthink for hundreds of tokens on the simplest user queries without improving performance. In contrast, large non-thinking models underthink, often falling short of much smaller thinking models. We further explore several methods to encourage optimal thinking, but find that these approaches often improve on one sub-benchmark at the expense of the other, highlighting the need for better unified and optimal models in the future.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 30 of its 32 references that are in the map. First posted on arXiv 18 Aug 2025.

Builds on

30 of 32 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Inverse Scaling in Test-Time Compute19 Jul
  2. 2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiency10 Jun
  3. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficient23 May
  4. 2025Let LRMs Break Free from Overthinking via Self-Braking Tuning20 May
  5. 2025FlashThink: An Early Exit Method For Efficient Reasoning20 May
  6. 2025AdaptThink: Reasoning Models Can Learn When to Think19 May
  7. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs30 Apr
  8. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning30 Apr
  9. 2025Exploring the System 1 Thinking Capability of Large Reasoning Models14 Apr
  10. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging26 Mar
  11. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models20 Mar
  12. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
4 more
  1. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks12 Feb
  2. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs30 Jan
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  4. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness16 Dec

Other references · 14

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  6. 2025Training Language Models to Reason Efficientlycited by 247
  7. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  8. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  9. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 109
  10. 2025Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversationscited by 96
  11. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  12. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
2 more
  1. 2025Walk Before You Run! Concise LLM Reasoning via Reinforcement Learningcited by 16
  2. 2025Arch-Router: Aligning LLM Routing with Human Preferencescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 24.

35 papers.

Sort:

By area:

  1. 2026How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Taskscited by 33
  2. 2025FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoningcited by 27
  3. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  4. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 19
  5. 2025When to Reason: Semantic Router for vLLMcited by 12
  6. 2025OckBench: Measuring the Efficiency of LLM Reasoningcited by 11
  7. 2025Do LLMs Really Need 10+ Thoughts for "Find the Time 1000 Days Later"? Towards Structural Understanding of LLM Overthinkingcited by 10
  8. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  9. 2026ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoningcited by 7
  10. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  11. 2025Pathways of Thoughts: Multi-Directional Thinking for Long-form Personalized Question Answeringcited by 5
  12. 2025Code-enabled language models can outperform reasoning models on diverse taskscited by 4
23 more
  1. 2025PBFuzz: Agentic Directed Fuzzing for PoV Generationcited by 4
  2. 2026Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoningcited by 4
  3. 2025A note on conditional PAC-efficient reasoning in large language model routingcited by 3
  4. 2026CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMscited by 3
  5. 2026Reasoning Shift: How Context Silently Shortens LLM Reasoningcited by 3
  6. 2026Reliable Chain-of-Thought via Prefix Consistencycited by 3
  7. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3
  8. 2025Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compressioncited by 2
  9. 2025Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmarkcited by 2
  10. 2025Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generationscited by 2
  11. 2026CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicinecited by 2
  12. 2026Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Taskscited by 1
  13. 2026Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planningcited by 1
  14. 2026Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding taskscited by 1
  15. 2026Conditional Performance Guarantee for Large Reasoning Modelscited by 0
  16. 2026A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategiescited by 0
  17. 2026Fuel Gauge: Estimating Chain-of-Thought Length Ahead of Time in Large Multimodal Modelscited by 0
  18. 2026Post-Reasoning: Improving the Performance of Non-Thinking Models at No Costcited by 0
  19. 2026Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoningcited by 0
  20. 2026ThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphscited by 0
  21. 2026Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichmentcited by 0
  22. 2026REChart: Reasoning-Efficient Chart Editing with Large Reasoning Modelscited by 0
  23. 2026Don't Overthink, Don't Underthink: Toward Adaptive Reasoning in Agentic AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models
  2. 2025Reasoning Models Can Be Effective Without Thinking
  3. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  4. 2025Explore Briefly, Then Decide: Mitigating LLM Overthinking via Cumulative Entropy Regulation
  5. 2024Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems
  6. 2026HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs
  7. 2026When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
  8. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving