mlsanityData to 14 Oct 2026

Two-Scale Latent Dynamics for Recurrent-Depth Transformers

arXiv 2509.23314 ↗First posted 27 Sep 2025cs.LGcited by 13

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recurrent-depth transformers scale test-time compute by iterating latent computations before emitting tokens. We study the geometry of these iterates and argue for a simple, two-scale operational picture: (i) within a looped block, updates act as small-scale refinements; (ii) across consecutive blocks, states undergo a larger-scale drift. Across training, our measurements show that loop steps become smaller and increasingly orthogonal to one another, indicating better local modeling of fine structure rather than merely pushing in a single direction. These dynamics motivate an early-exit mechanism based on the model's second-order difference in step-size, which we show is superior in terms of performance, stability and time-efficiency, when compared to the KL-divergence exit strategy of Geiping et al. and its naive first-order counterpart.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 6 of its 10 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

6 of 10 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Training Large Language Models to Reason in a Continuous Latent Space9 Dec

Other references · 5

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  2. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  3. 2025Hierarchical Reasoning Modelcited by 177
  4. 2024Looped Transformers for Length Generalizationcited by 99
  5. 2025Navigating the Latent Space Dynamics of Neural Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 33
  2. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 18
  3. 2026Training-Free Looped Transformerscited by 15
  4. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  5. 2026Hyperloop Transformerscited by 12
  6. 2026Loop the Loopies!cited by 8
  7. 2026LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scalingcited by 4
  8. 2026Looped World Modelscited by 3
  9. 2026Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformerscited by 1
  10. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 1
  11. 2026Reshape and Recur: Improving SSMs with Input Reshaping and Depth Recurrencecited by 0
  12. 2026Depth as Time in One-Step Generative Modelscited by 0
1 more
  1. 2026Looped Reasoning Finishes Earlier Than You Thinkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Looped Transformers as Optimizers
  2. 2026Dynamical phase selection controls compute scaling in looped transformers
  3. 2025Block-Recurrent Dynamics in Vision Transformers
  4. 2026Trading Depth for Time in Recurrent Transformers
  5. 2025Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
  6. 2026Looped Transformers with Source-Centered State Evolution
  7. 2026The Topological Trouble With Transformers
  8. 2026Stability and Generalization in Looped Transformers