mlsanityData to 14 Oct 2026

Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking

arXiv 2502.13842 ↗First posted 19 Feb 2025cs.CLcited by 57

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) face inherent performance bottlenecks under parameter constraints, particularly in processing critical tokens that demand complex reasoning. Empirical analysis reveals challenging tokens induce abrupt gradient spikes across layers, exposing architectural stress points in standard Transformers. Building on this insight, we propose Inner Thinking Transformer (ITT), which reimagines layer computations as implicit thinking steps. ITT dynamically allocates computation through Adaptive Token Routing, iteratively refines representations via Residual Thinking Connections, and distinguishes reasoning phases using Thinking Step Encoding. ITT enables deeper processing of critical tokens without parameter expansion. Evaluations across 162M-466M parameter models show ITT achieves 96.5% performance of a 466M Transformer using only 162M parameters, reduces training data by 43.2%, and outperforms Transformer/Loop variants in 11 benchmarks. By enabling elastic computation allocation during inference, ITT balances performance and efficiency through architecture-aware optimization of implicit thinking pathways.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 37 of its 58 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

37 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach7 Feb
  2. 2025s1: Simple test-time scaling31 Jan
  3. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability29 Nov
  4. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective31 Oct
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2024Mixtral of Expertscited by 1,823
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  9. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  10. 2018Universal Transformerscited by 647
  11. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  12. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
20 more
  1. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 234
  2. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  3. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  4. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  5. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  6. 2024Mixture of A Million Expertscited by 72
  7. 2020LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoningcited by 69
  8. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  9. 2024Beyond Model Adaptation at Test Time: A Surveycited by 59
  10. 2024Transformer Layers as Painterscited by 54
  11. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 35
  12. 2024Multi-Head Mixture-of-Expertscited by 31
  13. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Modelscited by 25
  14. 2024Distributional reasoning in LLMs: Parallel reasoning processes in multi-hop reasoningcited by 21
  15. 2024p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decaycited by 16
  16. 2024Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoningcited by 13
  17. 2024Loop Neural Networks for Parameter Sharingcited by 11
  18. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 8
  19. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 7
  20. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 30.

57 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  3. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  4. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  5. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 85
  6. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoningcited by 78
  7. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 70
  8. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  9. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 43
  10. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  11. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  12. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
38 more
  1. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 24
  2. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  3. 2025Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMscited by 16
  4. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 16
  5. 2025A Formal Comparison Between Chain of Thought and Latent Thoughtcited by 15
  6. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  7. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 12
  8. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Expertscited by 11
  9. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  10. 2025Exploring the System 1 Thinking Capability of Large Reasoning Modelscited by 10
  11. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMscited by 9
  12. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  13. 2026Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformationcited by 8
  14. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  15. 2025Efficient Pretraining Length Scalingcited by 6
  16. 2025Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysiscited by 6
  17. 2026Tiny Autoregressive Recursive Modelscited by 6
  18. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 5
  19. 2026ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inferencecited by 5
  20. 2026Pretraining with Token-Level Adaptive Latent Chain-of-Thoughtcited by 4
  21. 2026PonderLM-3: Adaptive Token-Wise Pondering with Differentiable Maskingcited by 4
  22. 2025SpiralThinker: Latent Reasoning through an Iterative Process with Text-Latent Interleavingcited by 3
  23. 2026Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Loopingcited by 3
  24. 2026The Topological Trouble With Transformerscited by 3
  25. 2026Semantic-Enriched Latent Visual Reasoningcited by 3
  26. 2025Catch Your Breath: Adaptive Computation for Self-Paced Sequence Productioncited by 2
  27. 2025Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understandingcited by 2
  28. 2026From Words to Amino Acids: Does the Curse of Depth Persist?cited by 2
  29. 2026MACRO: Markov Chain Routing of Transformer Layerscited by 2
  30. 2026T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning with Dynamic Routingcited by 2
  31. 2025DND: Boosting Large Language Models with Dynamic Nested Depthcited by 1
  32. 2025Flash Multi-Head Feed-Forward Networkcited by 1
  33. 2026Emergent Search and Backtracking in Latent Reasoning Modelscited by 1
  34. 2026Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulationcited by 1
  35. 2026Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don'tcited by 1
  36. 2026Programs-of-Layers in LLMs through the Lens of Cortical Areascited by 1
  37. 2026LA-Sign: Looped Transformers with Geometry-aware Alignment for Skeleton-based Sign Language Recognitioncited by 0
  38. 2026Adaptive Computation Depth via Learned Token Routing in Transformerscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Change of Thought: Adaptive Test-Time Computation
  2. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning
  3. 2025ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute
  4. 2025Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space
  5. 2026Transformers Provably Learn to Internalize Chain-of-Thought
  6. 2025Deep Thinking by Markov Chain of Continuous Thoughts
  7. 2026Thinking Deeper, Not Longer: Memory-Efficient Test-Time Reasoning with Depth-Recurrent Transformers for Compositional Generalization
  8. 2026Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens