mlsanityData to 7 Oct 2026

A Mechanistic Analysis of Looped Reasoning Language Models

arXiv 2604.11791 ↗First posted 13 Apr 2026cs.LG cs.AIcited by 27

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM's layers in the latent dimension, resulting in looped reasoning language models. Despite promising results, few works have investigated how their internal dynamics differ from those of standard feedforward models. In this paper, we conduct a mechanistic analysis of the latent states in looped language models, focusing in particular on how the stages of inference observed in feedforward models compare to those observed in looped ones. To this end, we analyze cyclic recurrence and show that for many of the studied models each layer in the cycle converges to a distinct fixed point; consequently, the recurrent block follows a consistent cyclic trajectory in the latent space. We provide evidence that as these fixed points are reached, attention-head behavior stabilizes, leading to constant behavior across recurrences. Empirically, we discover that recurrent blocks learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration. We study how recurrent block size, input injection, and normalization influence the emergence and stability of these cyclic fixed points. We believe these findings help translate mechanistic insights into practical guidance for architectural design.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 34 of its 51 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

34 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence10 Nov
  2. 2025Scaling Latent Reasoning via Looped Language Models29 Oct
  3. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts8 Oct
  4. 2025Less is More: Recursive Reasoning with Tiny Networks6 Oct
  5. 2025Hierarchical Reasoning Model26 Jun
  6. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformers24 Feb
  7. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach7 Feb

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  5. 2018Universal Transformerscited by 613
  6. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  7. 2020A Note on Over-Smoothing for Graph Neural Networkscited by 245
  8. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  9. 2021PonderNet: Learning to Pondercited by 156
  10. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  11. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  12. 2025Why do LLMs attend to the first token?cited by 122
15 more
  1. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  2. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  3. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  4. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 42
  5. 2023Sparse Universal Transformercited by 28
  6. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 28
  7. 2025Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformercited by 24
  8. 2025Continuous Thought Machinescited by 21
  9. 2025Softmax is 1/2-Lipschitz: A tight bound across all ℓp normscited by 18
  10. 2025Return of ChebNet: Understanding and Improving an Overlooked GNN on Long Range Taskscited by 12
  11. 2025Two-Scale Latent Dynamics for Recurrent-Depth Transformerscited by 11
  12. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  13. 2025Identifying and Evaluating Inactive Heads in Pretrained LLMscited by 8
  14. 2024Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Studycited by 4
  15. 2025gLSTM: Mitigating Over-Squashing by Increasing Storage Capacitycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 27.

27 papers.

Sort:

By area:

  1. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 21
  2. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  3. 2026Hyperloop Transformerscited by 11
  4. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 11
  5. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 11
  6. 2026Thinking with Looped Flowscited by 6
  7. 2026LT2: Linear-Time Looped Transformerscited by 5
  8. 2026Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Modelscited by 5
  9. 2026When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformerscited by 5
  10. 2026Looped Latent Attention: Cross-Loop KV Compression for Looped Transformerscited by 4
  11. 2026Loop the Loopies!cited by 4
  12. 2026Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?cited by 3
15 more
  1. 2026Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separationcited by 2
  2. 2026Mechanistic Analysis of Alignment Algorithms in Language Modelscited by 1
  3. 2026SMolLM: Small Language Models Learn Small Molecular Grammarcited by 0
  4. 2026Reshape and Recur: Improving SSMs with Input Reshaping and Depth Recurrencecited by 0
  5. 2026One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformercited by 0
  6. 2026Behaviour Is an Incomplete Measure of Reasoning Development: Cross-surface pre-arrival accessibility and the limits of developmental inference in a recurrent-depth reasonercited by 0
  7. 2026Steering Recurrent Reasoners at Inference Time with Readout Feedbackcited by 0
  8. 2026If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMscited by 0
  9. 2026Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformerscited by 0
  10. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 0
  11. 2026LoopICL: Looping a single transformer block to solve tabular taskscited by 0
  12. 2026Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes Itcited by 0
  13. 2026What Makes Recurrence Effective in Looped Language Models?cited by 0
  14. 2026LoopVL: Recurrent Visual Intelligencecited by 0
  15. 2026Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Doescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
  2. 2026Can We Break LLMs Out of Self-Loops? Fine-Grained Reasoning Control with Activation Steering
  3. 2026Closing the Loop: Practical Training Recipes for Looped Language Models
  4. 2026Circular Reasoning: Understanding Self-Reinforcing Loops in Large Reasoning Models
  5. 2026Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs
  6. 2026RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory
  7. 2026Dissecting Failure Dynamics in Large Language Model Reasoning
  8. 2026Enhancing reasoning accuracy in large language models during inference time