# Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

arXiv: https://arxiv.org/abs/2602.13517 · HTML: https://mlsanity.com/paper/2602.13517/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 13 Feb 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 29 of 29)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2502.03373/) (2025)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [Deep Think with Confidence](https://mlsanity.com/paper/2508.15260/) (2025)
- [Scalable Best-of-N Selection for Large Language Models via Self-Certainty](https://mlsanity.com/paper/2502.18581/) (2025)
- [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024)
- [Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs](https://mlsanity.com/paper/2505.00127/) (2025)
- [Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning](https://mlsanity.com/paper/2505.17813/) (2025)
- [Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning](https://mlsanity.com/paper/2508.09726/) (2025)
- [Archon: An Architecture Search Framework for Inference-Time Techniques](https://mlsanity.com/paper/2409.15254/) (2024)
- [The Limits of Inference Scaling Through Resampling](https://mlsanity.com/paper/2411.17501/) (2024)
- [Do Language Models Use Their Depth Efficiently?](https://mlsanity.com/paper/2505.13898/) (2025)
- [Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality](https://mlsanity.com/paper/2505.18227/) (2025)
- [Inverse Scaling in Test-Time Compute](https://mlsanity.com/paper/2507.14417/) (2025)
- [Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead](https://mlsanity.com/paper/2504.00294/) (2025)
- [Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective](https://mlsanity.com/paper/2505.12886/) (2025)
- [Shrinking the Generation-Verification Gap with Weak Verifiers](https://mlsanity.com/paper/2506.18203/) (2025)
- [First Finish Search: Efficient Test-Time Scaling in Large Language Models](https://mlsanity.com/paper/2505.18149/) (2025)
- [BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT](https://mlsanity.com/paper/2001.09309/) (2020)
- [Test-Time Scaling with Repeated Sampling Improves Multilingual Text Generation](https://mlsanity.com/paper/2505.21941/) (2025)

## Built on by (most cited first, 22 of 22)

- [Co-Evolving Skill Generation and Policy Optimization](https://mlsanity.com/paper/2606.08755/) (2026)
- [On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning](https://mlsanity.com/paper/2604.01702/) (2026)
- [Latent State Design for World Models under Sufficiency Constraints](https://mlsanity.com/paper/2605.01694/) (2026)
- [When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions](https://mlsanity.com/paper/2605.22873/) (2026)
- [Reasoning Models Know What's Important, and Encode It in Their Activations](https://mlsanity.com/paper/2604.18307/) (2026)
- [GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions](https://mlsanity.com/paper/2605.15764/) (2026)
- [Human vs Machine Mathematical Difficulty on Project Euler: An Experimental Analysis](https://mlsanity.com/paper/2606.21972/) (2026)
- [Improving Test-Time Scaling with Adaptive Looped Transformers](https://mlsanity.com/paper/2609.35748/) (2026)
- [Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models](https://mlsanity.com/paper/2605.01853/) (2026)
- [Stateful Reasoning via Insight Replay](https://mlsanity.com/paper/2605.14457/) (2026)
- [Effort as Ceiling, Not Dial: Reasoning Budget Does Not Modulate Cognitive Cost Alignment Between Humans and Large Reasoning Models](https://mlsanity.com/paper/2605.16938/) (2026)
- [Integrated and Cross-Architecture Interpretation of LLM Reasoning](https://mlsanity.com/paper/2605.28006/) (2026)
- [QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization](https://mlsanity.com/paper/2607.21793/) (2026)
- [ThinkReset: Learnable Intermediate Interface Construction for Bounded-Context Long-Horizon Reasoning](https://mlsanity.com/paper/2607.28642/) (2026)
- [How Hard Does It Think? Analyzing Step-Aware Reasoning Energy in LLM Chain-of-Thought Trajectories](https://mlsanity.com/paper/2607.28674/) (2026)
- [On the use of foundation models in cognitive science](https://mlsanity.com/paper/2608.07812/) (2026)
- [Expert-validated STEM QA](https://mlsanity.com/paper/2608.28591/) (2026)
- [Test-Time Scaling for Video Diffusion Models via Diagnosis-Guided Candidate Recycling](https://mlsanity.com/paper/2608.29322/) (2026)
- [Thinking effort aligns between humans and reasoning models in abductive reasoning](https://mlsanity.com/paper/2609.01867/) (2026)
- [S3: Spectral Null-Space Swap Makes Reasoning Models Efficient](https://mlsanity.com/paper/2609.37976/) (2026)
- [HeadEdit: Calibrating Language Model Behavior Through the Frozen Unembedding Matrix](https://mlsanity.com/paper/2610.01170/) (2026)
- [Where Is the "Self" of a Large Language Model? -- From Self-Index to Self-Reference](https://mlsanity.com/paper/2610.16717/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal "overthinking," leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens -- tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@n, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@n matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.