# Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

arXiv: https://arxiv.org/abs/2506.09250 · HTML: https://mlsanity.com/paper/2506.09250/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 2 of its 4 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 2 of 2)

- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity](https://mlsanity.com/paper/2506.06941/) (2025)

## Built on by (most cited first, 23 of 23)

- [Frontier LLMs Still Struggle with Simple Reasoning Tasks](https://mlsanity.com/paper/2507.07313/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning](https://mlsanity.com/paper/2507.10624/) (2025)
- [Are LLM Belief Updates Consistent with Bayes' Theorem?](https://mlsanity.com/paper/2507.17951/) (2025)
- [Lexical Hints of Accuracy in LLM Reasoning Chains](https://mlsanity.com/paper/2508.15842/) (2025)
- [A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap](https://mlsanity.com/paper/2506.18957/) (2025)
- [BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models](https://mlsanity.com/paper/2509.24210/) (2025)
- [How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective](https://mlsanity.com/paper/2508.01300/) (2025)
- [End-to-end PDDL Planning with Hardcoded and Dynamic Agents](https://mlsanity.com/paper/2512.09629/) (2025)
- [Reasoning Models Reason Well, Until They Don't](https://mlsanity.com/paper/2510.22371/) (2025)
- [CoreThink: A Symbolic Reasoning Layer to reason over Long Horizon Tasks with LLMs](https://mlsanity.com/paper/2509.00971/) (2025)
- [Rethinking the Illusion of Thinking](https://mlsanity.com/paper/2507.01231/) (2025)
- [Remapping and navigation of an embedding space via error minimization: a fundamental organizational principle of cognition in natural and artificial systems](https://mlsanity.com/paper/2601.14096/) (2026)
- [Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering](https://mlsanity.com/paper/2507.14406/) (2025)
- [The STAR-XAI Protocol: A Framework for Inducing and Verifying Agency, Reasoning, and Reliability in AI Agents](https://mlsanity.com/paper/2509.17978/) (2025)
- [Limits of Emergent Reasoning of Large Language Models in Agentic Frameworks for Deterministic Games](https://mlsanity.com/paper/2510.15974/) (2025)
- [Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis](https://mlsanity.com/paper/2602.10881/) (2026)
- [What properties of reasoning supervision are associated with improved downstream model quality?](https://mlsanity.com/paper/2605.13290/) (2026)
- [Learning to Reason Efficiently with A* Post-Training](https://mlsanity.com/paper/2605.24597/) (2026)
- [TRACE: An Operational Reasoning Schema for Auditable Agentic Commitments](https://mlsanity.com/paper/2607.12480/) (2026)
- [WordPolo: Evaluating Language Models Through Iterative Semantic Feedback](https://mlsanity.com/paper/2609.19006/) (2026)
- [A Systematic Study of Small Language Models on Abstract Reasoning Tasks](https://mlsanity.com/paper/2610.08680/) (2026)

## Abstract (arXiv)

Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit "accuracy collapse" on planning puzzles beyond certain complexity thresholds. We demonstrate that their findings primarily reflect experimental design limitations rather than fundamental reasoning failures. Our analysis reveals three critical issues: (1) Tower of Hanoi experiments risk exceeding model output token limits, with models explicitly acknowledging these constraints in their outputs; (2) The authors' automated evaluation framework fails to distinguish between reasoning failures and practical constraints, leading to misclassification of model capabilities; (3) Most concerningly, their River Crossing benchmarks include mathematically impossible instances for N > 5 due to insufficient boat capacity, yet models are scored as failures for not solving these unsolvable problems. When we control for these experimental artifacts, by requesting generating functions instead of exhaustive move lists, preliminary experiments across multiple models indicate high accuracy on Tower of Hanoi instances previously reported as complete failures. These findings highlight the importance of careful experimental design when evaluating AI reasoning capabilities.