mlsanityData to 7 Oct 2026

Looped Transformers as Programmable Computers

arXiv 2301.13196 ↗First posted 30 Jan 2023cs.LG cs.AIcited by 224

Not on the map. Why?

Cited by 50 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including embedding edit operations, non-linear functions, function calls, program counters, and conditional branches. Using these building blocks, we emulate a small instruction-set computer. This allows us to map iterative algorithms to programs that can be executed by a looped, 13-layer transformer. We show how this transformer, instructed by its input, can emulate a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Our work highlights the versatility of the attention mechanism, and demonstrates that even shallow transformers can execute full-fledged, general-purpose programs.

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

29 of 45 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  11. 2021Transformers in Vision: A Surveycited by 1,009
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
17 more
  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  4. 2018Universal Transformerscited by 613
  5. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  6. 2022Transformers Learn Shortcuts to Automatacited by 260
  7. 2021Thinking Like Transformerscited by 182
  8. 2022Language models show human-like content effects on reasoning taskscited by 171
  9. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  10. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  11. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  12. 2022Block-Recurrent Transformerscited by 107
  13. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  14. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  15. 2021Linear algebra with transformerscited by 62
  16. 2021Sliced Recursive Transformercited by 26
  17. 2018The universal approximation power of finite-width deep ReLU networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 27; 2024: 54; 2025: 42; 2026: 101.

224 papers.

Sort:

By area:

  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  2. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  3. 2023Augmented Language Models: a Surveycited by 398
  4. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  5. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  6. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  7. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  8. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  9. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  10. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  11. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  12. 2023LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasterscited by 127
38 more
  1. 2023Teaching Arithmetic to Small Transformerscited by 121
  2. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  3. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  4. 2023In-Context Convergence of Transformerscited by 112
  5. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  6. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  7. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  8. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  9. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  10. 2025Efficient Reasoning Models: A Surveycited by 93
  11. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  12. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  13. 2023Transformers as Support Vector Machinescited by 67
  14. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  15. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  16. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  17. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  18. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  19. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  20. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  21. 2023An Over-parameterized Exponential Regressioncited by 45
  22. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  23. 2024Dual Operating Modes of In-Context Learningcited by 42
  24. 2023Learning Transformer Programscited by 40
  25. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  26. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Modelscited by 39
  27. 2024An Information-Theoretic Analysis of In-Context Learningcited by 39
  28. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  29. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  30. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  31. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  32. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  33. 2024Simulation of Graph Algorithms with Looped Transformerscited by 29
  34. 2023Approximation Rate of the Transformer Architecture for Sequence Modelingcited by 28
  35. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  36. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 28
  37. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 28
  38. 2024Transformers on Markov Data: Constant Depth Sufficescited by 26

50 most cited of 224 All 224 (JSON)