mlsanityData to 7 Oct 2026

Transformer Feed-Forward Layers Are Key-Value Memories

arXiv 2012.14913 ↗First posted 29 Dec 2020cs.CLcited by 1,307

Not on the map. Why?

Cited by 93 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Feed-forward layers constitute two-thirds of a transformer model's parameters, yet their role in the network remains under-explored. We show that feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary. Our experiments show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones. The values complement the keys' input patterns by inducing output distributions that concentrate probability mass on tokens likely to appear immediately after each pattern, particularly in the upper layers. Finally, we demonstrate that the output of a feed-forward layer is a composition of its memories, which is subsequently refined throughout the model's layers via residual connections to produce the final output distribution.

Key facts. Cited by 1,307 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 39; 2023: 153; 2024: 314; 2025: 367; 2026: 421.

1,307 papers.

Sort:

By area:

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  4. 2022Mass-Editing Memory in a Transformercited by 976
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  8. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  9. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  11. 2021Fast Model Editing at Scalecited by 491
  12. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
38 more
  1. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  2. 2023Explainability for Large Language Models: A Surveycited by 331
  3. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  4. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  5. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  6. 2023Function Vectors in Large Language Modelscited by 292
  7. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  8. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  9. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  10. 2023Large Language Model Alignment: A Surveycited by 241
  11. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  12. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  13. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  14. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  15. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  16. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  17. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  18. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  19. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  20. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  21. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  22. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  23. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  24. 2023PMET: Precise Model Editing in a Transformercited by 170
  25. 2022Large Language Models with Controllable Working Memorycited by 169
  26. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  27. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  28. 2022A Review on Language Models as Knowledge Basescited by 156
  29. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  30. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  31. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  32. 2024How do Large Language Models Handle Multilingualism?cited by 148
  33. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  34. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  35. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  36. 2022Analyzing Transformers in Embedding Spacecited by 131
  37. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  38. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123

50 most cited of 1,307 All 1,307 (JSON)