mlsanityData to 7 Oct 2026

Memorization Capacity of Multi-Head Attention in Transformers

arXiv 2306.02010 ↗First posted 3 Jun 2023cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

Transformers have become the go-to architecture for language and vision tasks, yet their theoretical properties, especially memorization capacity, remain elusive. This paper investigates the memorization abilities of multi-head attention mechanisms, examining how many example sequences they can memorize, as a function of the number of heads and sequence length. Motivated by experimental findings on vision transformers, we introduce novel assumptions about the linear independence of input data, distinct from the commonly used general-position assumption. Under these assumptions, we demonstrate that an attention layer with H heads, dimension d, and context size n < d, featuring Theta(Hd2) parameters, can memorize Omega(Hn) examples. Our analysis sheds light on how different attention heads handle various example sequences, aided by the softmax operator's saturation property. We validate our findings through experiments on synthetic data.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 3 Jun 2023.

Builds on

32 of 48 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2022Transformers learn in-context by gradient descentcited by 714
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
20 more
  1. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  2. 2020Hopfield Networks is All You Needcited by 461
  3. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  4. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  5. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  6. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  7. 2022Vision Transformers provably learn spatial structurecited by 92
  8. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 88
  9. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
  10. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  11. 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 76
  12. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  13. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  14. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  15. 2020Provable Memorization via Deep Neural Networks using Sub-linear Parameterscited by 34
  16. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  17. 2020Network size and weights size for memorization with two-layers neural networkscited by 16
  18. 2021An Exponential Improvement on the Memorization Capacity of Deep Threshold Networkscited by 14
  19. 2020Memory capacity of neural networks with threshold and ReLU activationscited by 13
  20. 2022When Expressivity Meets Trainability: Fewer than n Neurons Can Workcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 19; 2025: 12; 2026: 4.

39 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2025How much do language models memorize?cited by 53
  3. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 46
  4. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 45
  5. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)cited by 39
  6. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  7. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  8. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  9. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  10. 2025Taming Knowledge Conflicts in Language Modelscited by 27
  11. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
  12. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
27 more
  1. 2024What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Taskscited by 19
  2. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  3. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14
  4. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 13
  5. 2024Next-token prediction capacity: general upper bounds and a lower bound for transformerscited by 11
  6. 2024On the Optimal Memorization Capacity of Transformerscited by 10
  7. 2024Understanding Addition and Subtraction in Transformerscited by 6
  8. 2024Global Convergence in Training Large-Scale Transformerscited by 5
  9. 2025MeMo: Towards Language Models with Associative Memory Mechanismscited by 5
  10. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  11. 2024On the Benefits of Rank in Attention Layerscited by 4
  12. 2025Memory Limitations of Prompt Tuning in Transformerscited by 4
  13. 2025The Effect of Attention Head Count on Transformer Approximationcited by 4
  14. 2023State space models can express n-gram languagescited by 3
  15. 2024Memorization in Attention-only Transformerscited by 2
  16. 2025Exact Sequence Interpolation with Transformerscited by 2
  17. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 2
  18. 2026On the Expressive Power of Floating-Point Transformerscited by 2
  19. 2024Empirical Capacity Model for Self-Attention Neural Networkscited by 1
  20. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Modelscited by 1
  21. 2025Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regressioncited by 1
  22. 2025Analyzing the Power of Chain of Thought through Memorization Capabilitiescited by 1
  23. 2026Geometric Factual Recall in Transformerscited by 1
  24. 2024Hierarchically Disentangled Recurrent Network for Factorizing System Dynamics of Multi-scale Systems: An application on Hydrological Systemscited by 0
  25. 2025Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Datacited by 0
  26. 2026Approximation Error Upper and Lower Bounds for H"older Class with Transformerscited by 0
  27. 2026On the Capability and Limitation of Hard Promptcited by 0