mlsanityData to 14 Oct 2026

Uncovering mesa-optimization algorithms in Transformers

arXiv 2309.05858 ↗First posted 11 Sep 2023cs.LG cs.AIcited by 100

Not on the map. Why?

Abstract (arXiv)

Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without being explicitly trained to do so. The origins of this phenomenon are still poorly understood. Here we analyze a series of Transformer models trained to perform synthetic sequence prediction tasks, and discover that standard next-token prediction error minimization gives rise to a subsidiary learning algorithm that adjusts the model as new inputs are revealed. We show that this process corresponds to gradient-based optimization of a principled objective function, which leads to strong generalization performance on unseen sequences. Our findings explain in-context learning as a product of autoregressive loss minimization and inform the design of new optimization-based Transformer layers.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Builds on 67 of its 123 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

67 of 123 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2016Layer Normalizationcited by 5,669
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2020Rethinking Attention with Performerscited by 1,601
  6. 2018World Modelscited by 1,513
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  12. 2022Transformers learn in-context by gradient descentcited by 732
  13. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  14. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  15. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  16. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  17. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  18. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  19. 2020Hopfield Networks is All You Needcited by 475
  20. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  21. 2016Learning to reinforcement learncited by 377
  22. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  23. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  24. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  25. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  26. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  27. 2022Transformers Learn Shortcuts to Automatacited by 268
  28. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  29. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  30. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  31. 2023Looped Transformers as Programmable Computerscited by 237
  32. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  33. 2016Using Fast Weights to Attend to the Recent Pastcited by 170
  34. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  35. 2014Difference Target Propagationcited by 148
  36. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  37. 2019Training Neural Networks with Local Error Signalscited by 136
  38. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 115

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 38; 2025: 25; 2026: 28.

100 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  4. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  5. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  6. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  7. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  8. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  9. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  10. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  11. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 64
  12. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
38 more
  1. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 59
  2. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  3. 2024ICLR: In-Context Learning of Representationscited by 50
  4. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 49
  5. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 49
  6. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 47
  7. 2024Transformers are Universal In-context Learnerscited by 43
  8. 2025Log-Linear Attentioncited by 40
  9. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  10. 2024Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Taskscited by 34
  11. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  12. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  13. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 25
  14. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 24
  15. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  16. 2023In-context Learning and Gradient Descent Revisitedcited by 20
  17. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  18. 2023Structured World Representations in Maze-Solving Transformerscited by 16
  19. 2023Towards Understanding How Transformers Learn In-context Through a Representation Learning Lenscited by 15
  20. 2026Memory Caching: RNNs with Growing Memorycited by 15
  21. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 14
  22. 2024Attention as a Hypernetworkcited by 13
  23. 2025Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuningcited by 13
  24. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 12
  25. 2023In-Context Learning for MIMO Equalization Using Transformer-Based Sequence Modelscited by 11
  26. 2024On the Role of Depth and Looping for In-Context Learning with Task Diversitycited by 11
  27. 2025Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergencecited by 10
  28. 2024Towards Understanding the Universality of Transformers for Next-Token Predictioncited by 9
  29. 2024In-context learning and Occam's razorcited by 8
  30. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 8
  31. 2025Rethinking Invariance in In-context Learningcited by 8
  32. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 8
  33. 2025Selective Induction Heads: How Transformers Select Causal Structures In Contextcited by 8
  34. 2023Metalearning Continual Learning Algorithmscited by 7
  35. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  36. 2024When can transformers compositionally generalize in-context?cited by 7
  37. 2024On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recoverycited by 6
  38. 2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6

50 most cited of 100 All 100 (JSON)