mlsanityData to 7 Oct 2026

Evaluating the Ripple Effects of Knowledge Editing in Language Models

arXiv 2307.12976 ↗First posted 24 Jul 2023cs.CLcited by 238

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations. This has led to the development of various editing methods that allow updating facts encoded by the model. Evaluation of these methods has primarily focused on testing whether an individual fact has been successfully injected, and if similar predictions for other subjects have not changed. Here we argue that such evaluation is limited, since injecting one fact (e.g. ``Jack Depp is the son of Johnny Depp'') introduces a ``ripple effect'' in the form of additional facts that the model needs to update (e.g.``Jack Depp is the sibling of Lily-Rose Depp''). To address this issue, we propose a novel set of evaluation criteria that consider the implications of an edit on related facts. Using these criteria, we then construct RippleEdits, a diagnostic benchmark of 5K factual edits, capturing a variety of types of ripple effects. We evaluate prominent editing methods on RippleEdits, showing that current methods fail to introduce consistent changes in the model's knowledge. In addition, we find that a simple in-context editing baseline obtains the best scores on our benchmark, suggesting a promising research direction for model editing.

Key facts. Cited by 238 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

39 of 52 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2022Mass-Editing Memory in a Transformercited by 976
27 more
  1. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  4. 2021Editing Factual Knowledge in Language Modelscited by 548
  5. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  7. 2021Fast Model Editing at Scalecited by 491
  8. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  9. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  10. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  11. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  12. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  13. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  14. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  15. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  16. 2022Prompting GPT-3 To Be Reliablecited by 258
  17. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  18. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  19. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  20. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  21. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  22. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  23. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  24. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  25. 2023Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmarkcited by 62
  26. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 56
  27. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 91; 2025: 68; 2026: 60.

238 papers.

Sort:

By area:

  1. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  2. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  3. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  4. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  5. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  6. 2023PMET: Precise Model Editing in a Transformercited by 170
  7. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  8. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  9. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  10. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 101
  11. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  12. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 92
38 more
  1. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  2. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  3. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  4. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  5. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  6. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  7. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 56
  8. 2024Synthetic continued pretrainingcited by 56
  9. 2024A Unified Framework for Model Editingcited by 54
  10. 2024Rebuilding ROME : Resolving Model Collapse during Sequential Model Editingcited by 50
  11. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  12. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  13. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  14. 2023Can We Edit Multimodal Large Language Models?cited by 44
  15. 2024Learning to Edit: Aligning LLMs with Knowledge Editingcited by 43
  16. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 43
  17. 2024Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learningcited by 40
  18. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  19. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
  20. 2023Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Educationcited by 38
  21. 2024Knowledge Graph Enhanced Large Language Model Editingcited by 38
  22. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  23. 2024Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Factscited by 36
  24. 2024Uncovering Overfitting in Large Language Model Editingcited by 36
  25. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 35
  26. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  27. 2024DeepEdit: Knowledge Editing as Decoding with Constraintscited by 33
  28. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 33
  29. 2024VLKEB: A Large Vision-Language Model Knowledge Editing Benchmarkcited by 30
  30. 2024Logically Consistent Language Models via Neuro-Symbolic Integrationcited by 30
  31. 2023Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarkscited by 29
  32. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  33. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  34. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  35. 2023Emptying the Ocean with a Spoon: Should We Edit Models?cited by 27
  36. 2024Perturbation-Restrained Sequential Model Editingcited by 27
  37. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  38. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 26

50 most cited of 238 All 238 (JSON)