mlsanityData to 7 Oct 2026

Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models

arXiv 2301.04213 ↗First posted 10 Jan 2023cs.LG cs.AI cs.CLcited by 237

Not on the map. Why?

Cited by 29 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models learn a great quantity of factual information during pretraining, and recent work localizes this information to specific model weights like mid-layer MLP weights. In this paper, we find that we can change how a fact is stored in a model by editing weights that are in a different location than where existing methods suggest that the fact is stored. This is surprising because we would expect that localizing facts to specific model parameters would tell us where to manipulate knowledge in models, and this assumption has motivated past work on model editing methods. Specifically, we show that localization conclusions from representation denoising (also known as Causal Tracing) do not provide any insight into which model MLP layer would be best to edit in order to override an existing stored fact with a new one. This finding raises questions about how past work relies on Causal Tracing to select which model layers to edit. Next, we consider several variants of the editing problem, including erasing and amplifying facts. For one of our editing problems, editing performance does relate to localization results from representation denoising, but we find that which layer we edit is a far better predictor of performance. Our results suggest, counterintuitively, that better mechanistic understanding of how pretrained language models work may not always translate to insights about how to best change their behavior. Our code is available at https://github.com/google/belief-localization

Key facts. Cited by 237 papers in the map (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

34 of 61 references.

References · 34

  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2022Mass-Editing Memory in a Transformercited by 976
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  8. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  9. 2021Editing Factual Knowledge in Language Modelscited by 548
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  11. 2021Fast Model Editing at Scalecited by 491
  12. 2022Memory-Based Model Editing at Scalecited by 443
22 more
  1. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  2. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  3. 2020Modifying Memories in Transformer Modelscited by 222
  4. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  5. 2020Compositional Explanations of Neuronscited by 172
  6. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  7. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  8. 2020Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectorscited by 119
  9. 2020Rewriting a Deep Generative Modelcited by 105
  10. 2017Interpreting Deep Visual Representations via Network Dissectioncited by 103
  11. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  12. 2021An Interpretability Illusion for BERTcited by 94
  13. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  14. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  15. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  16. 2021Editing a classifier by rewriting its prediction rulescited by 64
  17. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  18. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  19. 2022Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspacescited by 25
  20. 2021Of Non-Linearity and Commutativity in BERTcited by 16
  21. 2020Explaining Neural Networks by Decoding Layer Activationscited by 13
  22. 2022Local Relighting of Real Scenescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 38; 2024: 92; 2025: 58; 2026: 49.

237 papers.

Sort:

By area:

  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2023Steering Language Models With Activation Engineeringcited by 824
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  6. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  7. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  8. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  9. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  10. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  11. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  12. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
38 more
  1. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  2. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  3. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  4. 2024How to use and interpret activation patchingcited by 191
  5. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  6. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  7. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  8. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  9. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  10. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  11. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 101
  12. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  13. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  14. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  15. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  16. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  17. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  18. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  19. 2025Do Multilingual LLMs Think In English?cited by 75
  20. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  21. 2023Massive Editing for Large Language Models via Meta Learningcited by 62
  22. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  23. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  24. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  25. 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
  26. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  27. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  28. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  29. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  30. 2023Can We Edit Multimodal Large Language Models?cited by 44
  31. 2024Learning to Edit: Aligning LLMs with Knowledge Editingcited by 43
  32. 2024Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMscited by 39
  33. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  34. 2024Locating and Editing Factual Associations in Mambacited by 35
  35. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  36. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 34
  37. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  38. 2024Activation Scaling for Steering and Interpreting Language Modelscited by 33

50 most cited of 237 All 237 (JSON)