mlsanityData to 7 Oct 2026

Knowledge Neurons in Pretrained Transformers

arXiv 2104.08696 ↗First posted 18 Apr 2021cs.CLcited by 590

Not on the map. Why?

Cited by 36 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons. Specifically, we examine the fill-in-the-blank cloze task for BERT. Given a relational fact, we propose a knowledge attribution method to identify the neurons that express the fact. We find that the activation of such knowledge neurons is positively correlated to the expression of their corresponding facts. In our case studies, we attempt to leverage knowledge neurons to edit (such as update, and erase) specific factual knowledge without fine-tuning. Our results shed light on understanding the storage of knowledge within pretrained Transformers. The code is available at https://github.com/Hunter-DDM/knowledge-neurons.

Key facts. Cited by 590 papers in the map (data through 7 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

30 of 38 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,088
18 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  6. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  7. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  8. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  10. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  11. 2019Revealing the Dark Secrets of BERTcited by 300
  12. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  13. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  14. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  15. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  16. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  17. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 124
  18. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 31; 2023: 109; 2024: 169; 2025: 142; 2026: 127.

590 papers.

Sort:

By area:

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Mass-Editing Memory in a Transformercited by 976
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  6. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2023Erasing Concepts from Diffusion Modelscited by 530
  9. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  11. 2021Fast Model Editing at Scalecited by 491
  12. 2022Memory-Based Model Editing at Scalecited by 443
38 more
  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  3. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  4. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  5. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  6. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  7. 2022A Survey of Machine Unlearningcited by 298
  8. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  9. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  10. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  11. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  12. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  13. 2023Large Language Model Alignment: A Surveycited by 241
  14. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  15. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  16. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  17. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  18. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  19. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  20. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  21. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  22. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  23. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  24. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  25. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  26. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  27. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  28. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  29. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  30. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  31. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  32. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  33. 2022Analyzing Transformers in Embedding Spacecited by 131
  34. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  35. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 128
  36. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  37. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  38. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118

50 most cited of 590 All 590 (JSON)