# Looped Transformers for Length Generalization

arXiv: https://arxiv.org/abs/2409.15647 · HTML: https://mlsanity.com/paper/2409.15647/

## Key facts

- Cited by 99 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2024.

## Built on by (most cited first, 30 of 99)

- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation](https://mlsanity.com/paper/2507.10524/) (2025)
- [Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA](https://mlsanity.com/paper/2410.20672/) (2024)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation](https://mlsanity.com/paper/2602.11451/) (2026)
- [Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence](https://mlsanity.com/paper/2511.07384/) (2025)
- [On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding](https://mlsanity.com/paper/2410.01405/) (2024)
- [Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner](https://mlsanity.com/paper/2510.03206/) (2025)
- [Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges](https://mlsanity.com/paper/2502.01612/) (2025)
- [Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers](https://mlsanity.com/paper/2606.18206/) (2026)
- [Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts](https://mlsanity.com/paper/2510.07358/) (2025)
- [Parallel Loop Transformer for Efficient Test-Time Computation Scaling](https://mlsanity.com/paper/2510.24824/) (2025)
- [Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models](https://mlsanity.com/paper/2605.26733/) (2026)
- [Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning](https://mlsanity.com/paper/2502.08482/) (2025)
- [Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking](https://mlsanity.com/paper/2502.20129/) (2025)
- [Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs](https://mlsanity.com/paper/2507.07996/) (2025)
- [A Formal Comparison Between Chain of Thought and Latent Thought](https://mlsanity.com/paper/2509.25239/) (2025)
- [LoopViT: Scaling Visual ARC with Looped Transformers](https://mlsanity.com/paper/2602.02156/) (2026)
- [Training-Free Looped Transformers](https://mlsanity.com/paper/2605.23872/) (2026)
- [ELT: Elastic Looped Transformers for Visual Generation](https://mlsanity.com/paper/2604.09168/) (2026)
- [Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models](https://mlsanity.com/paper/2605.07721/) (2026)
- [Extrapolation by Association: Length Generalization Transfer in Transformers](https://mlsanity.com/paper/2506.09251/) (2025)
- [Two-Scale Latent Dynamics for Recurrent-Depth Transformers](https://mlsanity.com/paper/2509.23314/) (2025)
- [Solve the Loop: Attractor Models for Language and Reasoning](https://mlsanity.com/paper/2605.12466/) (2026)
- [SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion](https://mlsanity.com/paper/2602.11698/) (2026)
- [DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation](https://mlsanity.com/paper/2506.14202/) (2025)
- [Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning](https://mlsanity.com/paper/2502.04667/) (2025)
- [On the Residual Scaling of Looped Transformers: Stability and Transferability](https://mlsanity.com/paper/2606.18524/) (2026)
- [Skip a Layer or Loop It? Learning Program-of-Layers in LLMs](https://mlsanity.com/paper/2606.06574/) (2026)
- [MeSH: Memory-as-State-Highways for Recursive Transformers](https://mlsanity.com/paper/2510.07739/) (2025)

## Abstract (arXiv)

Recent work has shown that Transformers trained from scratch can successfully solve various arithmetic and algorithmic tasks, such as adding numbers and computing parity. While these Transformers generalize well on unseen inputs of the same length, they struggle with length generalization, i.e., handling inputs of unseen lengths. In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization. We focus on tasks with a known iterative solution, involving multiple iterations of a RASP-L operation - a length-generalizable operation that can be expressed by a finite-sized Transformer. We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks.