mlsanityData to 7 Oct 2026

How Transformers Get Rich: Approximation and Dynamics Analysis

arXiv 2410.11474 ↗First posted 15 Oct 2024cs.LG math.OC stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage et al., 2021) identified a ``rich'' in-context mechanism known as induction head, contrasting with ``lazy'' n-gram models that overlook long-range dependencies. In this work, we provide both approximation and dynamics analyses of how transformers implement induction heads. In the em approximation analysis, we formalize both standard and generalized induction head mechanisms, and examine how transformers can efficiently implement them, with an emphasis on the distinct role of each transformer submodule. For the em dynamics analysis, we study the training dynamics on a synthetic mixed target, composed of a 4-gram and an in-context 2-gram component. This controlled setting allows us to precisely characterize the entire training process and uncover an em abrupt transition from lazy (4-gram) to rich (induction head) mechanisms as training progresses.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

51 of 67 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2022Transformers learn in-context by gradient descentcited by 714
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
38 more
  1. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  2. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  3. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  4. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  5. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  6. 2023Looped Transformers as Programmable Computerscited by 224
  7. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  8. 2021Thinking Like Transformerscited by 182
  9. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  10. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  11. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  12. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  13. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  14. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  15. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  16. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  17. 2023In-Context Convergence of Transformerscited by 112
  18. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  19. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  20. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  21. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  22. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  23. 2024Transformers, parallel computation, and logarithmic depthcited by 75
  24. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 71
  25. 2023Transformers as Support Vector Machinescited by 67
  26. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  27. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  28. 2023Transformers learn through gradual rank increasecited by 52
  29. 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
  30. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 46
  31. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  32. 2024One-layer transformers fail to solve the induction heads taskcited by 30
  33. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  34. 2024Transformers on Markov Data: Constant Depth Sufficescited by 26
  35. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  36. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 21
  37. 2024What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Taskscited by 19
  38. 2021Optimization-Based Separations for Neural Networkscited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  2. 2025An explainable transformer circuit for compositional generalizationcited by 8
  3. 2024KV Shifting Attention Enhances Language Modelingcited by 7
  4. 2025What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformerscited by 5
  5. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  6. 2026Ability Transfer and Recovery via Modularized Parameters Localizationcited by 0
  7. 2026Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Inductioncited by 0