# Recurrent Looped Transformer

arXiv: https://arxiv.org/abs/2610.07591 · HTML: https://mlsanity.com/paper/2610.07591/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 19 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 17 of 17)

- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Reasoning with Latent Thoughts: On the Power of Looped Transformers](https://mlsanity.com/paper/2502.17416/) (2025)
- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [The Illusion of State in State-Space Models](https://mlsanity.com/paper/2404.08819/) (2024)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [Block-Recurrent Transformers](https://mlsanity.com/paper/2203.07852/) (2022)
- [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://mlsanity.com/paper/1804.09849/) (2018)
- [Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues](https://mlsanity.com/paper/2411.12537/) (2024)
- [PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space](https://mlsanity.com/paper/2509.23184/) (2025)
- [Addressing Some Limitations of Transformers with Feedback Memory](https://mlsanity.com/paper/2002.09402/) (2020)
- [DeepLoop: Depth Scaling for Looped Transformers](https://mlsanity.com/paper/2607.13491/) (2026)
- [T^2MLR: Transformer with Temporal Middle-Layer Recurrence](https://mlsanity.com/paper/2607.15178/) (2026)
- [The Recurrent Transformer: Greater Effective Depth and Efficient Decoding](https://mlsanity.com/paper/2604.21215/) (2026)
- [Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior](https://mlsanity.com/paper/2605.26797/) (2026)
- [Full-bandwidth transformer](https://mlsanity.com/paper/2608.08888/) (2026)

## Abstract (arXiv)

State tracking requires an update at every input, but the depth a Transformer applies to each token is fixed regardless of sequence length. We introduce the Recurrent Looped Transformer (RLT), which splits its layers between a parallel causal encoder and a recurrent decoder. At each token, the decoder merges the encoder output with the previous token's final decoder state, so the computation path grows with sequence length at a fixed per-token cost. On six algorithmic tasks, we compare five splits of eight layers with an eight-layer Transformer over three seeds. Trained on at most 40 bits, two RLT splits generalize parity to 256 bits with 100% accuracy in every seed, while the Transformer stays at chance. On swap-based S5 permutation tracking at eight times the training length, RLT reaches 97% final-state accuracy versus under 1% for the Transformer, and accuracy increases with decoder depth. On modular arithmetic beyond the training lengths, RLT reaches up to 93% versus 33% for the Transformer. Ablations show that these gains depend on the feedback: removing it drops parity and swap-based S5 to chance at every split. Updating the feedback once per four-token chunk lets known tokens in a chunk run in parallel and keeps 64-bit parity at 99%, while permutation tracking depends on per-token feedback: chunking lowers length-64 swap-based S5 from 100% to 20%.