mlsanityData to 14 Oct 2026

Scaling Laws for Associative Memories

arXiv 2310.02984 ↗First posted 4 Oct 2023stat.ML cs.AI cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models. We derive precise scaling laws with respect to sample size and parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms. We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

25 of 43 references.

References · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2022Toy Models of Superpositioncited by 971
  7. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  9. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  10. 2020Hopfield Networks is All You Needcited by 475
  11. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  12. 2021Explaining Neural Scaling Lawscited by 363
13 more
  1. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  3. 2016Dense Associative Memory for Pattern Recognitioncited by 236
  4. 2022Memorizing Transformerscited by 226
  5. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  6. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  7. 2023The Quantization Model of Neural Scalingcited by 138
  8. 2022A Solvable Model of Neural Scaling Lawscited by 116
  9. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  10. 2019Augmenting Self-attention with Persistent Memorycited by 93
  11. 2021Learning Curve Theorycited by 85
  12. 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 38
  13. 2023Multiperiodic Processes: Ergodic Sources with a Sublinear Entropycited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 10; 2026: 11.

35 papers.

Sort:

By area:

  1. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  2. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  3. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  4. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  5. 2024A Simple Model of Inference Scaling Lawscited by 36
  6. 2024Large Language Models as Markov Chainscited by 35
  7. 2025Taming Knowledge Conflicts in Language Modelscited by 30
  8. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 23
  9. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  10. 2024Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codescited by 19
  11. 2024Learning Associative Memories with Gradient Descentcited by 16
  12. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 15
23 more
  1. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 15
  2. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 13
  3. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
  4. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 9
  5. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  6. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  7. 2025Provable Benefits of In-Tool Learning for Large Language Modelscited by 4
  8. 2025From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesiscited by 3
  9. 2026Sharp Capacity Thresholds in Linear Associative Memory: From Top-1 Retrieval to Tail-Average Learningcited by 3
  10. 2024Semantically-correlated memories in a dense associative modelcited by 2
  11. 2024Memorization in Attention-only Transformerscited by 2
  12. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2
  13. 2025Attention with Trained Embeddings Provably Selects Important Tokenscited by 1
  14. 2026Factual recall in linear associative memories: sharp asymptotics and mechanistic insightscited by 1
  15. 2026Geometric Factual Recall in Transformerscited by 1
  16. 2026Fine-Tuning Dynamics of In-Context Factual Recall in Transformerscited by 1
  17. 2026MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformerscited by 1
  18. 2024Rethinking Associative Memory Mechanism in Induction Headcited by 0
  19. 2026Sinkhorn Based Associative Memory Retrieval Using Spherical Hellinger Kantorovich Dynamicscited by 0
  20. 2026Single-Layer MeMo as a Randomized Hamming-Kernel Classifiercited by 0
  21. 2026Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalizationcited by 0
  22. 2026Divergence controls entropy in distillationcited by 0
  23. 2026Exact Scaling Laws for a Random-Features Bigram Modelcited by 0