# Two-Scale Latent Dynamics for Recurrent-Depth Transformers

arXiv: https://arxiv.org/abs/2509.23314 · HTML: https://mlsanity.com/paper/2509.23314/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 6 of 6)

- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [Hierarchical Reasoning Model](https://mlsanity.com/paper/2506.21734/) (2025)
- [Looped Transformers for Length Generalization](https://mlsanity.com/paper/2409.15647/) (2024)
- [Navigating the Latent Space Dynamics of Neural Models](https://mlsanity.com/paper/2505.22785/) (2025)

## Built on by (most cited first, 13 of 13)

- [A Mechanistic Analysis of Looped Reasoning Language Models](https://mlsanity.com/paper/2604.11791/) (2026)
- [Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning](https://mlsanity.com/paper/2605.21488/) (2026)
- [Training-Free Looped Transformers](https://mlsanity.com/paper/2605.23872/) (2026)
- [Solve the Loop: Attractor Models for Language and Reasoning](https://mlsanity.com/paper/2605.12466/) (2026)
- [Hyperloop Transformers](https://mlsanity.com/paper/2604.21254/) (2026)
- [Loop the Loopies!](https://mlsanity.com/paper/2607.16051/) (2026)
- [LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling](https://mlsanity.com/paper/2606.18023/) (2026)
- [Looped World Models](https://mlsanity.com/paper/2606.18208/) (2026)
- [Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers](https://mlsanity.com/paper/2609.16665/) (2026)
- [Improving Test-Time Scaling with Adaptive Looped Transformers](https://mlsanity.com/paper/2609.35748/) (2026)
- [Reshape and Recur: Improving SSMs with Input Reshaping and Depth Recurrence](https://mlsanity.com/paper/2605.16048/) (2026)
- [Depth as Time in One-Step Generative Models](https://mlsanity.com/paper/2610.03626/) (2026)
- [Looped Reasoning Finishes Earlier Than You Think](https://mlsanity.com/paper/2610.18174/) (2026)

## Abstract (arXiv)

Recurrent-depth transformers scale test-time compute by iterating latent computations before emitting tokens. We study the geometry of these iterates and argue for a simple, two-scale operational picture: (i) within a looped block, updates act as small-scale refinements; (ii) across consecutive blocks, states undergo a larger-scale drift. Across training, our measurements show that loop steps become smaller and increasingly orthogonal to one another, indicating better local modeling of fine structure rather than merely pushing in a single direction. These dynamics motivate an early-exit mechanism based on the model's second-order difference in step-size, which we show is superior in terms of performance, stability and time-efficiency, when compared to the KL-divergence exit strategy of Geiping et al. and its naive first-order counterpart.