A Mechanistic Study of Transformers Training Dynamics
Not on the map. Why?
Abstract (arXiv)
Large-scale pretraining of transformers has been central to the success of foundation models. However, the scale of those models limits our understanding of the mechanisms at play during optimization. In this work, we study the training dynamics of transformers in a controlled and interpretable setting. On the sparse modular addition task, we demonstrate that specialized attention circuits, called clustering heads, can be implemented during gradient descent to solve the problem. Our experiments show that such pathways naturally emerge during training. By monitoring the evolution of tokens via a visual sandbox, we uncover a two-stage learning and the occurrences of loss spikes due to the high curvature of normalization layers. Our findings provide several insights into patterns observed in more practical settings, such as the pretraining of large language models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 67 of its 100 references that are in the map. First posted on arXiv 31 Oct 2024.
Builds on
67 of 100 references.
References · 67
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
- 2019Root Mean Square Layer Normalizationcited by 1,425
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
38 more
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
- 2022In-context Learning and Induction Headscited by 930
- 2020On Layer Normalization in the Transformer Architecturecited by 906
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
- 2023Trained Transformers Learn Linear Models In-Contextcited by 331
- 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
- 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
- 2016Using the Output Embedding to Improve Language Modelscited by 266
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
- 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
- 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
- 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
- 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
- 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
- 2023Birth of a Transformer: A Memory Viewpointcited by 154
- 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
- 2023The emergence of clusters in self-attention dynamicscited by 133
- 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
- 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
- 2023Teaching Arithmetic to Small Transformerscited by 121
- 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
- 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
- 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
- 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
- 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
- 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
- 2022Vision Transformers provably learn spatial structurecited by 92
- 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
50 most cited of 67 All 67 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.