mlsanityData to 14 Oct 2026

Looped Transformers for Length Generalization

arXiv 2409.15647 ↗First posted 24 Sep 2024cs.LGcited by 99

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work has shown that Transformers trained from scratch can successfully solve various arithmetic and algorithmic tasks, such as adding numbers and computing parity. While these Transformers generalize well on unseen inputs of the same length, they struggle with length generalization, i.e., handling inputs of unseen lengths. In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization. We focus on tasks with a known iterative solution, involving multiple iterations of a RASP-L operation - a length-generalizable operation that can be expressed by a finite-sized Transformer. We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks.

Key facts. Cited by 99 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Sep 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 28; 2026: 67.

99 papers.

Sort:

By area:

  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  2. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  3. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  4. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  5. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  6. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  7. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 36
  8. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  9. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  10. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 28
  11. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 24
  12. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 24
38 more
  1. 2026Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Modelscited by 21
  2. 2025Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoningcited by 19
  3. 2025Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Trackingcited by 18
  4. 2025Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMscited by 16
  5. 2025A Formal Comparison Between Chain of Thought and Latent Thoughtcited by 15
  6. 2026LoopViT: Scaling Visual ARC with Looped Transformerscited by 15
  7. 2026Training-Free Looped Transformerscited by 15
  8. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 14
  9. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 14
  10. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  11. 2025Two-Scale Latent Dynamics for Recurrent-Depth Transformerscited by 13
  12. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  13. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 12
  14. 2025DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretationcited by 11
  15. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10
  16. 2026On the Residual Scaling of Looped Transformers: Stability and Transferabilitycited by 10
  17. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMscited by 9
  18. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  19. 2025Quantitative Bounds for Length Generalization in Transformerscited by 8
  20. 2026Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Modelscited by 8
  21. 2026Loop the Loopies!cited by 8
  22. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thoughtcited by 7
  23. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 7
  24. 2026LT2: Linear-Time Looped Transformerscited by 7
  25. 2025Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfactioncited by 6
  26. 2025Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoningcited by 6
  27. 2026Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddingscited by 6
  28. 2026LoopQ: Quantization for Looped Language Modelscited by 6
  29. 2026When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformerscited by 6
  30. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 5
  31. 2026Step-resolved data attribution for looped transformerscited by 5
  32. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 5
  33. 2024Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Studycited by 4
  34. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Modelscited by 4
  35. 2026Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stoppingcited by 4
  36. 2026Allocating Recurrent Compute in Looped Language Modelscited by 4
  37. 2025Intra-Layer Recurrence in Transformers for Language Modelingcited by 3
  38. 2025The Imitation Game: Turing Machine Imitator is Length Generalizable Reasonercited by 3

50 most cited of 99 All 99 (JSON)