mlsanityData to 7 Oct 2026

Inductive Biases and Variable Creation in Self-Attention Mechanisms

arXiv 2110.10090 ↗First posted 19 Oct 2021cs.LG stat.MLcited by 144

Not on the map. Why?

Abstract (arXiv)

Self-attention, an architectural motif designed to model long-range interactions in sequential data, has driven numerous recent breakthroughs in natural language processing and beyond. This work provides a theoretical analysis of the inductive biases of self-attention modules. Our focus is to rigorously establish which functions and long-range dependencies self-attention blocks prefer to represent. Our main result shows that bounded-norm Transformer networks "create sparse variables": a single self-attention head can represent a sparse function of the input sequence, with sample complexity scaling only logarithmically with the context length. To support our analysis, we present synthetic experiments to probe the sample complexity of learning sparse Boolean functions with Transformers.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Builds on 43 of its 79 references that are in the map. First posted on arXiv 19 Oct 2021.

Builds on

43 of 79 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2016Layer Normalizationcited by 5,579
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
31 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  3. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  8. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  11. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  12. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  13. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  14. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  15. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  16. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  17. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  18. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  19. 2019Recurrent Independent Mechanismscited by 219
  20. 2020The Lipschitz Constant of Self-Attentioncited by 171
  21. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  22. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  23. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  24. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  25. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  26. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  27. 2019Generalization bounds for deep convolutional neural networkscited by 58
  28. 2021On the Expressive Power of Self-Attention Matricescited by 44
  29. 2020A Mathematical Theory of Attentioncited by 43
  30. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  31. 2021Approximating How Single Head Attention Learnscited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 42; 2024: 41; 2025: 26; 2026: 23.

144 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023Large Language Modelscited by 774
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  6. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  7. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  8. 2022Transformers Learn Shortcuts to Automatacited by 260
  9. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  10. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  11. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  12. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
38 more
  1. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  2. 2023Representational Strengths and Limitations of Transformerscited by 132
  3. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  4. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  5. 2022Vision Transformers provably learn spatial structurecited by 92
  6. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
  7. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  8. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  9. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  10. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  11. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 73
  12. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  13. 2022Formal Algorithms for Transformerscited by 70
  14. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  15. 2023Transformers as Support Vector Machinescited by 67
  16. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  17. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  18. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  19. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  20. 2023On the Role of Attention in Prompt-tuningcited by 49
  21. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  22. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 46
  23. 2023An Over-parameterized Exponential Regressioncited by 45
  24. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  25. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  26. 2023Memorization Capacity of Multi-Head Attention in Transformerscited by 39
  27. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  28. 2024An Information-Theoretic Analysis of In-Context Learningcited by 39
  29. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  30. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  31. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  32. 2024From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformerscited by 34
  33. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  34. 2024Large Language Models as Markov Chainscited by 33
  35. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  36. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  37. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  38. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30

50 most cited of 144 All 144 (JSON)