mlsanityData to 7 Oct 2026

Towards Understanding Inductive Bias in Transformers: A View From Infinity

arXiv 2402.05173 ↗First posted 7 Feb 2024cs.LG cond-mat.dis-nn stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

We study inductive bias in Transformers in the infinitely over-parameterized Gaussian process limit and argue transformers tend to be biased towards more permutation symmetric functions in sequence space. We show that the representation theory of the symmetric group can be used to give quantitative analytical predictions when the dataset is symmetric to permutations between tokens. We present a simplified transformer block and solve the model at the limit, including accurate predictions for the learning curves and network outputs. We show that in common setups, one can derive tight bounds in the form of a scaling law for the learnability as a function of the context length. Finally, we argue WikiText dataset, does indeed possess a degree of permutation symmetry.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 54 of its 82 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

54 of 82 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2018On the Spectral Bias of Neural Networkscited by 1,321
  7. 2016Group Equivariant Convolutional Networkscited by 1,154
  8. 2022In-context Learning and Induction Headscited by 930
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  11. 2022Transformers learn in-context by gradient descentcited by 714
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
38 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2017Deep Neural Networks as Gaussian Processescited by 617
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  4. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  5. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  6. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  7. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  8. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  9. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  10. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  11. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  12. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  13. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  14. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  15. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  16. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  17. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  18. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  19. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  20. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  21. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  22. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  23. 2020Neural Kernels Without Tangentscited by 56
  24. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  25. 2020Is SGD a Bayesian sampler? Well, almostcited by 50
  26. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  27. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 36
  28. 2021Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphscited by 35
  29. 2020Deep kernel processescited by 34
  30. 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
  31. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  32. 2023The Exact Sample Complexity Gain from Invariances for Kernel Regressioncited by 27
  33. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  34. 2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
  35. 2021Unified field theoretical approach to deep and recurrent neuronal networkscited by 20
  36. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  37. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  38. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 15

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 5.

13 papers.

Sort:

By area:

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  2. 2024Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformerscited by 9
  3. 2025Pool Me Wisely: On the Effect of Pooling in Transformer-Based Modelscited by 6
  4. 2025Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexitycited by 4
  5. 2024Demystifying Spectral Bias on Real-World Datacited by 3
  6. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  7. 2026Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Lawscited by 2
  8. 2025Adaptive Morph-Patch Transformer for Aortic Vessel Segmentationcited by 1
  9. 2026Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformerscited by 1
  10. 2025Customizing the Inductive Biases of Softmax Attention using Structured Matricescited by 0
  11. 2025Traj-Transformer: Diffusion Models with Transformer for GPS Trajectory Generationcited by 0
  12. 2026GIBLy: Improving 3D Semantic Segmentation through an Architecture-Agnostic Lightweight Geometric Inductive Bias Layercited by 0
1 more
  1. 2026On the Necessity of Attention-FFN Split in Vision Transformerscited by 0