mlsanityData to 14 Oct 2026

Attention Approximates Sparse Distributed Memory

arXiv 2111.05498 ↗First posted 10 Nov 2021cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well. Here, we show that Transformer Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 33 of its 86 references that are in the map. First posted on arXiv 10 Nov 2021.

Builds on

33 of 86 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2020Rethinking Attention with Performerscited by 1,601
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  12. 2020On Layer Normalization in the Transformer Architecturecited by 922
21 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 816
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  3. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  4. 2020Hopfield Networks is All You Needcited by 475
  5. 2020Query-Key Normalization for Transformerscited by 280
  6. 2016Dense Associative Memory for Pattern Recognitioncited by 236
  7. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  8. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  9. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  10. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 112
  11. 2017Building machines that adapt and compute like brainscited by 99
  12. 2019Augmenting Self-attention with Persistent Memorycited by 93
  13. 2019Shaping Belief States with Generative Environment Models for RLcited by 66
  14. 2015Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in Neocortexcited by 45
  15. 2016Associative Long Short-Term Memorycited by 37
  16. 2018The Sparse Manifold Transformcited by 36
  17. 2018The Kanerva Machine: A Generative Distributed Memorycited by 14
  18. 2018Learning Attractor Dynamics for Generative Memorycited by 6
  19. 2019Do Transformer Attention Heads Provide Transparency in Abstractive Summarization?cited by 6
  20. 2020Product Kanerva Machines: Factorized Bayesian Memorycited by 5
  21. 202050 years since the Marr, Ito, and Albus models of the cerebellumcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 8; 2024: 7; 2025: 2; 2026: 4.

29 papers.

Sort:

By area:

  1. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  2. 2021A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challengescited by 69
  3. 2022Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Modelscited by 55
  4. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  5. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
  6. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 24
  7. 2023Assessing Knowledge Editing in Language Models via Relation Perspectivecited by 15
  8. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 15
  9. 2022BayesPCN: A Continually Learnable Predictive Coding Associative Memorycited by 14
  10. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13
  11. 2023Sparse Distributed Memory is a Continual Learnercited by 11
  12. 2022O-ViT: Orthogonal Vision Transformercited by 10
17 more
  1. 2024Bridging Associative Memory and Probabilistic Modelingcited by 5
  2. 2022Should attention be all we need? The epistemic and ethical implications of unification in machine learningcited by 4
  3. 2022On the Relationship Between Variational Inference and Auto-Associative Memorycited by 4
  4. 2023Inverse distance weighting attentioncited by 4
  5. 2023Large Language Models are In-context Teachers for Knowledge Reasoningcited by 4
  6. 2024An In-depth Investigation of Sparse Rate Reduction in Transformer-like Modelscited by 4
  7. 2024Associative memory inspires improvements for in-context learning using a novel attention residual stream architecturecited by 3
  8. 2022Memory-Augmented Graph Neural Networks: A Brain-Inspired Reviewcited by 2
  9. 2024Associative Memories in the Feature Spacecited by 2
  10. 2023Hopfield-Enhanced Deep Neural Networks for Artifact-Resilient Brain State Decodingcited by 1
  11. 2024On Design Choices in Similarity-Preserving Sparse Randomized Embeddingscited by 1
  12. 2025From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformerscited by 1
  13. 2026MultiHashFormer: Hash-based Generative Language Modelscited by 1
  14. 2023A Framework for Analyzing Cross-correlators using Price's Theorem and Piecewise-Linear Decompositioncited by 0
  15. 2025Attention as Conditioning: What Classical Learning Theory Predicts About Linear Transformerscited by 0
  16. 2026Controlled Dynamics Attractor Transformercited by 0
  17. 2026When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformerscited by 0