# Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers

arXiv: https://arxiv.org/abs/2604.07822 · HTML: https://mlsanity.com/paper/2604.07822/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 8 of 8)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Scaling Latent Reasoning via Looped Language Models](https://mlsanity.com/paper/2510.25741/) (2025)
- [Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer](https://mlsanity.com/paper/2507.02199/) (2025)
- [Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts](https://mlsanity.com/paper/2509.26314/) (2025)
- [Lessons from Studying Two-Hop Latent Reasoning](https://mlsanity.com/paper/2411.16353/) (2024)

## Built on by (most cited first, 30 of 37)

- [Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers](https://mlsanity.com/paper/2606.18206/) (2026)
- [Training-Free Looped Transformers](https://mlsanity.com/paper/2605.23872/) (2026)
- [Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models](https://mlsanity.com/paper/2605.07721/) (2026)
- [Solve the Loop: Attractor Models for Language and Reasoning](https://mlsanity.com/paper/2605.12466/) (2026)
- [Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers](https://mlsanity.com/paper/2606.31779/) (2026)
- [Hyperloop Transformers](https://mlsanity.com/paper/2604.21254/) (2026)
- [Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts](https://mlsanity.com/paper/2607.20519/) (2026)
- [Think Shallow, Solve Deep: Controlling Recurrent Dynamics for Reliable Test-Time Depth](https://mlsanity.com/paper/2608.18222/) (2026)
- [Loop the Loopies!](https://mlsanity.com/paper/2607.16051/) (2026)
- [When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers](https://mlsanity.com/paper/2607.20594/) (2026)
- [Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching](https://mlsanity.com/paper/2608.09444/) (2026)
- [Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets](https://mlsanity.com/paper/2608.11233/) (2026)
- [Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models](https://mlsanity.com/paper/2606.24898/) (2026)
- [DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning](https://mlsanity.com/paper/2607.00341/) (2026)
- [What Survives When You Compress a Recursive Reasoner for the Edge?](https://mlsanity.com/paper/2606.26488/) (2026)
- [The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner](https://mlsanity.com/paper/2608.03263/) (2026)
- [Looped Language Models Improve Compositional Tool Calling](https://mlsanity.com/paper/2608.18171/) (2026)
- [Reason Through the Latent! Making Latent Visual Reasoning Necessary](https://mlsanity.com/paper/2609.06746/) (2026)
- [Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models](https://mlsanity.com/paper/2609.27373/) (2026)
- [SMolLM: Small Language Models Learn Small Molecular Grammar](https://mlsanity.com/paper/2605.06322/) (2026)
- [An Asymptotic Theory of Chain-of-Thought in In-Context Learning](https://mlsanity.com/paper/2606.03217/) (2026)
- [Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable Memory](https://mlsanity.com/paper/2606.20737/) (2026)
- [LayerNorm as Implicit Gain Control in Looped Transformers](https://mlsanity.com/paper/2607.10681/) (2026)
- [Recursive Vision Language Models for General Symbolic Reasoning](https://mlsanity.com/paper/2608.01534/) (2026)
- [Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling](https://mlsanity.com/paper/2609.09691/) (2026)
- [Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers](https://mlsanity.com/paper/2609.19521/) (2026)
- [Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning](https://mlsanity.com/paper/2609.19878/) (2026)
- [Write Back the Delta: Revisiting the Same Tokens with Fresh Representations](https://mlsanity.com/paper/2609.32457/) (2026)
- [Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers](https://mlsanity.com/paper/2609.33144/) (2026)
- [Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It](https://mlsanity.com/paper/2609.36585/) (2026)

## Abstract (arXiv)

We study implicit reasoning, i.e. the ability to combine knowledge or rules within a single forward pass. While transformer-based large language models store substantial factual knowledge and rules, they often fail to compose this knowledge for implicit multi-hop reasoning, suggesting a lack of compositional generalization over their parametric knowledge. To address this limitation, we study recurrent-depth transformers, which enables iterative computation over the same transformer layers. We investigate two compositional generalization challenges under the implicit reasoning scenario: systematic generalization, i.e. combining knowledge that is never used for compositions during training, and depth extrapolation, i.e. generalizing from limited reasoning depth (e.g. training on up to 5-hop) to deeper compositions (e.g. 10-hop). Through controlled studies with models trained from scratch, we show that while vanilla transformers struggle with both generalization challenges, recurrent-depth transformers can effectively make such generalization. For systematic generalization, we find that this ability emerges through a three-stage grokking process, transitioning from memorization to in-distribution generalization and finally to systematic generalization, supported by mechanistic analysis. For depth extrapolation, we show that generalization beyond training depth can be unlocked by scaling inference-time recurrence, with more iterations enabling deeper reasoning. We further study how training strategies affect extrapolation, providing guidance on training recurrent-depth transformers, and identify a key limitation, overthinking, where excessive recurrence degrades predictions and limits generalization to very deep compositions.