mlsanityData to 7 Oct 2026

Locating and Editing Factual Associations in GPT

arXiv 2202.05262 ↗First posted 10 Feb 2022cs.CL cs.LGcited by 2,609

Not on the map. Why?

Cited by 250 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at https://rome.baulab.info/

Key facts. Cited by 2,609 papers in the map (data through 7 Oct 2026). Builds on 32 of its 59 references that are in the map. First posted on arXiv 10 Feb 2022.

Builds on

32 of 59 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
20 more
  1. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  2. 2021Editing Factual Knowledge in Language Modelscited by 548
  3. 2021Fast Model Editing at Scalecited by 491
  4. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  5. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  6. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  7. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  8. 2013Direct and Indirect Effectscited by 276
  9. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  10. 2020Modifying Memories in Transformer Modelscited by 222
  11. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  12. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  13. 2020How Context Affects Language Models' Factual Predictionscited by 181
  14. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  15. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  16. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  17. 2018Generating Informative and Diverse Conversational Responses via Adversarial Information Maximizationcited by 114
  18. 2020Rewriting a Deep Generative Modelcited by 105
  19. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  20. 2021Of Non-Linearity and Commutativity in BERTcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 23; 2023: 244; 2024: 588; 2025: 687; 2026: 1067.

2,609 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2022Mass-Editing Memory in a Transformercited by 976
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  10. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  11. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  12. 2024KAN: Kolmogorov-Arnold Networkscited by 634
38 more
  1. 2023Erasing Concepts from Diffusion Modelscited by 530
  2. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  3. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  5. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  7. 2022Memory-Based Model Editing at Scalecited by 443
  8. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  9. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  10. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  11. 2023Unified Concept Editing in Diffusion Modelscited by 377
  12. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  13. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  14. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  15. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  16. 2023Explainability for Large Language Models: A Surveycited by 331
  17. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  18. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  19. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  20. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  21. 2023Language Models Represent Space and Timecited by 306
  22. 2023In-Context Learning Creates Task Vectorscited by 302
  23. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  24. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 298
  25. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  26. 2023Function Vectors in Large Language Modelscited by 292
  27. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  28. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  29. 2023Label-Free Concept Bottleneck Modelscited by 282
  30. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  31. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  32. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  33. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  34. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  35. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  36. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  37. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  38. 2023Large Language Model Alignment: A Surveycited by 241

50 most cited of 2,609 All 2,609 (JSON)