mlsanityData to 7 Oct 2026

Looped Transformers are Better at Learning Learning Algorithms

arXiv 2311.12424 ↗First posted 21 Nov 2023cs.LG cs.NEcited by 135

Not on the map. Why?

Cited by 41 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have demonstrated effectiveness in in-context solving data-fitting problems from various (latent) models, as reported by Garg et al. However, the absence of an inherent iterative structure in the transformer architecture presents a challenge in emulating the iterative algorithms, which are commonly employed in traditional machine learning methods. To address this, we propose the utilization of looped transformer architecture and its associated training methodology, with the aim of incorporating iterative characteristics into the transformer architectures. Experimental results suggest that the looped transformer achieves performance comparable to the standard transformer in solving various data-fitting problems, while utilizing less than 10% of the parameter count.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

48 of 62 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
36 more
  1. 2022In-context Learning and Induction Headscited by 930
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  3. 2022Transformers learn in-context by gradient descentcited by 714
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  6. 2019Deep Equilibrium Modelscited by 603
  7. 2014OpenML: networked science in machine learningcited by 523
  8. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  9. 2021Transformers Can Do Bayesian Inferencecited by 338
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  11. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  12. 2020Deep Learning Techniques for Inverse Problems in Imagingcited by 290
  13. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  14. 2022Transformers Learn Shortcuts to Automatacited by 260
  15. 2023Looped Transformers as Programmable Computerscited by 224
  16. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  17. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  18. 2020Multiscale Deep Equilibrium Modelscited by 149
  19. 2023Teaching Arithmetic to Small Transformerscited by 121
  20. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  21. 2022Block-Recurrent Transformerscited by 107
  22. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  23. 2020Monotone operator equilibrium networkscited by 100
  24. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 71
  25. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  26. 2021Linear algebra with transformerscited by 62
  27. 2018Trellis Networks for Sequence Modelingcited by 50
  28. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  29. 2019R-Transformer: Recurrent Neural Network Enhanced Transformercited by 39
  30. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  31. 2022Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networkscited by 27
  32. 2023CausalLM is not optimal for in-context learningcited by 26
  33. 2022What is my math transformer doing? -- Three results on interpretability and generalizationcited by 19
  34. 2023Can Transformers Learn to Solve Problems Recursively?cited by 17
  35. 2022A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithmscited by 8
  36. 2022Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithmscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 35; 2026: 86.

135 papers.

Sort:

By area:

  1. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  2. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  3. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  4. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  5. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  6. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  7. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  8. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  9. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 49
  10. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  11. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  12. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
38 more
  1. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 41
  2. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 28
  3. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
  4. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  5. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 23
  6. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  7. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 21
  8. 2025Continuous Thought Machinescited by 21
  9. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 21
  10. 2024On the Inductive Bias of Stacking Towards Improving Reasoningcited by 18
  11. 2026Generative Recursive Reasoningcited by 16
  12. 2025Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMscited by 14
  13. 2025A Formal Comparison Between Chain of Thought and Latent Thoughtcited by 14
  14. 2026LoopViT: Scaling Visual ARC with Looped Transformerscited by 14
  15. 2026Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Modelscited by 14
  16. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
  17. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  18. 2026Training-Free Looped Transformerscited by 12
  19. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 12
  20. 2024AlgoFormer: An Efficient Transformer Framework with Algorithmic Structurescited by 11
  21. 2024On the Role of Depth and Looping for In-Context Learning with Task Diversitycited by 11
  22. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 11
  23. 2026Hyperloop Transformerscited by 11
  24. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 11
  25. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 11
  26. 2024Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Pathcited by 10
  27. 2025Dr.LLM: Dynamic Layer Routing in LLMscited by 10
  28. 2026Residual Context Diffusion Language Modelscited by 9
  29. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 9
  30. 2026Stability and Generalization in Looped Transformerscited by 9
  31. 2024Transformers Handle Endogeneity in In-Context Linear Regressioncited by 8
  32. 2025Fast and Simplex: 2-Simplicial Attention in Tritoncited by 8
  33. 2025Deep Thinking by Markov Chain of Continuous Thoughtscited by 8
  34. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  35. 2024ALTA: Compiler-Based Analysis of Transformerscited by 7
  36. 2025AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modelingcited by 7
  37. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Modelscited by 7
  38. 2026Tiny Recursive Reasoning with Mamba-2 Attention Hybridcited by 7

50 most cited of 135 All 135 (JSON)