# Think Shallow, Solve Deep: Controlling Recurrent Dynamics for Reliable Test-Time Depth

arXiv: https://arxiv.org/abs/2608.18222 · HTML: https://mlsanity.com/paper/2608.18222/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 18 Aug 2026.

## Builds on (most cited first, 27 of 27)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Neural GPUs Learn Algorithms](https://mlsanity.com/paper/1511.08228/) (2015)
- [Ripser: efficient computation of Vietoris-Rips persistence barcodes](https://mlsanity.com/paper/1908.02518/) (2019)
- [Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks](https://mlsanity.com/paper/2106.04537/) (2021)
- [Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs](https://mlsanity.com/paper/2505.00127/) (2025)
- [Recurrent Relational Networks](https://mlsanity.com/paper/1711.08028/) (2017)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [Representation Topology Divergence: A Method for Comparing Neural Network Representations](https://mlsanity.com/paper/2201.00058/) (2021)
- [Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers](https://mlsanity.com/paper/2604.07822/) (2026)
- [Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics](https://mlsanity.com/paper/1905.12080/) (2019)
- [Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models](https://mlsanity.com/paper/2605.26733/) (2026)
- [Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning](https://mlsanity.com/paper/2605.21488/) (2026)
- [Stability and Generalization in Looped Transformers](https://mlsanity.com/paper/2604.15259/) (2026)
- [Improved memory in recurrent neural networks with sequential non-normal dynamics](https://mlsanity.com/paper/1905.13715/) (2019)
- [On Logical Extrapolation for Mazes with Recurrent and Implicit Networks](https://mlsanity.com/paper/2410.03020/) (2024)
- [Transient amplification in balanced neural networks](https://mlsanity.com/paper/2011.08215/) (2020)
- [NeuralSolver: Learning Algorithms For Consistent and Efficient Extrapolation Across General Tasks](https://mlsanity.com/paper/2402.15393/) (2024)
- [Dynamical Systems Theory Behind a Hierarchical Reasoning Model](https://mlsanity.com/paper/2603.22871/) (2026)

## Built on by (most cited first, 9 of 9)

- [Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers](https://mlsanity.com/paper/2609.16665/) (2026)
- [Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models](https://mlsanity.com/paper/2609.19934/) (2026)
- [When Recursive Models Finish Computing](https://mlsanity.com/paper/2609.26487/) (2026)
- [Improving Test-Time Scaling with Adaptive Looped Transformers](https://mlsanity.com/paper/2609.35748/) (2026)
- [Loop Dropout: Regularizing Shared Updates in Looped Language Models](https://mlsanity.com/paper/2609.34218/) (2026)
- [Scaling RNA Foundation Models through Iterative Refinement](https://mlsanity.com/paper/2610.13106/) (2026)
- [Fitting Every Hint Is Not Enough: Premature Floods in Recurrent Algorithmic Reasoners](https://mlsanity.com/paper/2610.14982/) (2026)
- [Resample the Answer: Where to Spend Test-Time Compute in Recursive Reasoning](https://mlsanity.com/paper/2610.16427/) (2026)
- [LoopREG: Representation Supervision Enables Test-Time Recurrent Depth in Diffusion Transformers](https://mlsanity.com/paper/2610.17877/) (2026)

## Abstract (arXiv)

Recurrent-depth reasoners aim to solve harder problems by iterating their update longer at test time, but additional iterations can improve, preserve, or degrade an answer. We show that a measurable property of the trained operator, its finite-time dynamical regime (estimated as settling, marginal, or drifting), indicates which of these occurs. We give a sufficient condition for depth-safety: once an operator's per-step displacement is small relative to the decoder margin, the decoded answer cannot change under further iterations. Empirically, on algorithmic tasks trained from 800 unaugmented examples per difficulty tier, settling operators do not degrade with added depth, and on some tasks convert it into higher accuracy on harder unseen instances (Sudoku, 0.19 to 0.34 past the training horizon). A single terminal fixed-point objective moves the regime and the depth behavior together: removing it induces drift and removes the gains, and adding it to a generic recurrence yields depth-safe extrapolation on carry propagation. We give four operational criteria for useful test-time depth, use them to catalogue failure modes, and, as a consistency check, apply the same measurements to Huginn-3.5B, which falls in the non-settling family.