mlsanityData to 14 Oct 2026

Understanding Factual Recall in Transformers via Associative Memories

arXiv 2412.06538 ↗First posted 9 Dec 2024cs.LG cs.CL cs.ITcited by 51

Not on the map. Why?

Abstract (arXiv)

Large language models have demonstrated an impressive ability to perform factual recall. Prior work has found that transformers trained on factual recall tasks can store information at a rate proportional to their parameter count. In our work, we show that shallow transformers can use a combination of associative memories to obtain such near optimal storage capacity. We begin by proving that the storage capacities of both linear and MLP associative memories scale linearly with parameter count. We next introduce a synthetic factual recall task, and prove that a transformer with a single layer of self-attention followed by an MLP can obtain 100% accuracy on the task whenever either the total number of self-attention parameters or MLP parameters scales (up to log factors) linearly with the number of facts. In particular, the transformer can trade off between using the value matrices or the MLP as an associative memory to store the dataset of facts. We complement these expressivity results with an analysis of the gradient flow trajectory of a simplified linear attention model trained on our factual recall task, where we show that the model exhibits sequential learning behavior.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 13 of its 50 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

13 of 50 references.

References · 13

  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2020Hopfield Networks is All You Needcited by 475
  3. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  4. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 47
  5. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  6. 2023Memorization Capacity of Multi-Head Attention in Transformerscited by 40
  7. 2021Approximating How Single Head Attention Learnscited by 38
  8. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 32
  9. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 31
  10. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 21
  11. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 15
  12. 2024Next-token prediction capacity: general upper bounds and a lower bound for transformerscited by 11
1 more
  1. 2024On the Optimal Memorization Capacity of Transformerscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 27.

51 papers.

Sort:

By area:

  1. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  2. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  3. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 37
  4. 2025Taming Knowledge Conflicts in Language Modelscited by 30
  5. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  6. 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 16
  7. 2025Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thoughtcited by 16
  8. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 15
  9. 2026Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memorycited by 14
  10. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  11. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
  12. 2025Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformerscited by 10
38 more
  1. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  2. 2026Why Fine-Tuning Encourages Hallucinations and How to Fix Itcited by 7
  3. 2026T^2MLR: Transformer with Temporal Middle-Layer Recurrencecited by 7
  4. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
  5. 2025Model-Aware Tokenizer Transfercited by 6
  6. 2025Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobianscited by 5
  7. 2025On the Emergence of Induction Heads for In-Context Learningcited by 5
  8. 2026MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformerscited by 5
  9. 2026Universal Redundancies in Time Series Foundation Modelscited by 5
  10. 2025Mamba Knockout for Unraveling Factual Information Flowcited by 4
  11. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  12. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  13. 2024Associative memory inspires improvements for in-context learning using a novel attention residual stream architecturecited by 3
  14. 2025Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformerscited by 3
  15. 2025Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalizationcited by 3
  16. 2026Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledgecited by 3
  17. 2026Sharp Capacity Thresholds in Linear Associative Memory: From Top-1 Retrieval to Tail-Average Learningcited by 3
  18. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  19. 2025How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalizationcited by 1
  20. 2026Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architecturescited by 1
  21. 2026Factual recall in linear associative memories: sharp asymptotics and mechanistic insightscited by 1
  22. 2026Geometric Factual Recall in Transformerscited by 1
  23. 2026The Future of Facts: Tracing the Factual Generation-Verification Gapcited by 1
  24. 2026Fine-Tuning Dynamics of In-Context Factual Recall in Transformerscited by 1
  25. 2026MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformerscited by 1
  26. 2026The Rank the Task Demands: A Causal Rank Law for Matrix Memories Trained on Group Compositioncited by 1
  27. 2025`Generalization is hallucination' through the lens of tensor completionscited by 0
  28. 2025Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectivenesscited by 0
  29. 2025Testing Transformer Learnability on the Arithmetic Sequence of Rooted Treescited by 0
  30. 2026Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformerscited by 0
  31. 2026TIDE: Every Layer Knows the Token Beneath the Contextcited by 0
  32. 2026A theoretical model for task routing in mixture-of-expert transformerscited by 0
  33. 2026The Privacy-Hallucination Tradeoff in Differentially Private Language Modelscited by 0
  34. 2026When the Gradient Sees Rank: Provable Necessity, Causal Recruitment, and Composition in Trained Matrix Memoriescited by 0
  35. 2026Circuit-Diff: Factual Edit-based Intervention Method for Localizing Knowledge in Attribution Graphscited by 0
  36. 2026What Pretraining and Midtraining Make Learnable from Rewards?cited by 0
  37. 2026Constant-Memory Recall: Learned Associations in a Fixed Matrix Statecited by 0
  38. 2026Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalizationcited by 0

50 most cited of 51 All 51 (JSON)