Explaining Grokking in Transformers through the Lens of Inductive Bias
Not on the map. Why?
Abstract (arXiv)
We investigate grokking in transformers through the lens of inductive bias: dispositions arising from architecture or optimization that let the network prefer one solution over another. We first show that architectural choices such as the position of Layer Normalization (LN) strongly modulates grokking speed. This modulation is explained by isolating how LN on specific pathways shapes shortcut-learning and attention entropy. Subsequently, we study how different optimization settings modulate grokking, inducing distinct interpretations of previously proposed controls such as readout scale. Particularly, we find that using readout scale as a control for lazy training can be confounded by learning rate and weight decay in our setting. Accordingly, we show that features evolve continuously throughout training, suggesting grokking in transformers can be more nuanced than a lazy-to-rich transition of the learning regime. Finally, we show how generalization predictably emerges with feature compressibility in grokking, across different modulators of inductive bias. Our code is released at https://tinyurl.com/y52u3cad.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 6 Feb 2026.
Builds on
39 of 52 references.
References · 39
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2016Layer Normalizationcited by 5,579
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2019Root Mean Square Layer Normalizationcited by 1,425
- 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2022In-context Learning and Induction Headscited by 930
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
- 2020On Layer Normalization in the Transformer Architecturecited by 906
27 more
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2017High-dimensional dynamics of generalization error in neural networkscited by 238
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
- 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
- 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
- 2023Explaining grokking through circuit efficiencycited by 105
- 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
- 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
- 2023Grokking modular arithmeticcited by 77
- 2024Knowledge Circuits in Pretrained Transformerscited by 60
- 2025The Curse of Depth in Large Language Modelscited by 58
- 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
- 2023Unifying Grokking and Double Descentcited by 49
- 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 32
- 2020Temperature check: theory and practice for training models with softmax-cross-entropy lossescited by 31
- 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
- 2025Feature Learning beyond the Lazy-Rich Dichotomy: Insights from Representational Geometrycited by 9
- 2025Normalization in Attention Dynamicscited by 9
- 2025Deep sequence models tend to memorize geometrically; it is unclear whycited by 9
- 2022InBiaseD: Inductive Bias Distillation to Improve Generalization and Robustness through Shape-awarenesscited by 5
- 2024Towards Exact Computation of Inductive Biascited by 5
- 2025Teasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networkscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.