mlsanityData to 14 Oct 2026

Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark

arXiv 2305.17553 ↗First posted 27 May 2023cs.CL cs.AI cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

Recent model editing techniques promise to mitigate the problem of memorizing false or outdated associations during LLM training. However, we show that these techniques can introduce large unwanted side effects which are not detected by existing specificity benchmarks. We extend the existing CounterFact benchmark to include a dynamic component and dub our benchmark CounterFact+. Additionally, we extend the metrics used for measuring specificity by a principled KL divergence-based metric. We use this improved benchmark to evaluate recent model editing techniques and find that they suffer from low specificity. Our findings highlight the need for improved specificity benchmarks that identify and prevent unwanted side effects.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

16 of 19 references.

References · 16

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2022Mass-Editing Memory in a Transformercited by 1,001
  8. 2019How Can We Know What Language Models Know?cited by 959
  9. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  11. 2021Editing Factual Knowledge in Language Modelscited by 553
  12. 2021Fast Model Editing at Scalecited by 498
4 more
  1. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  2. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  3. 2020Modifying Memories in Transformer Modelscited by 224
  4. 2020How Context Affects Language Models' Factual Predictionscited by 181

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 24; 2025: 14; 2026: 11.

62 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 254
  2. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  4. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 231
  5. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  6. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  7. 2024Uncovering Overfitting in Large Language Model Editingcited by 39
  8. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 34
  9. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  10. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  11. 2023Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarkscited by 29
  12. 2023Propagating Knowledge Updates to LMs Through Distillationcited by 28
38 more
  1. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  2. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  3. 2023Emptying the Ocean with a Spoon: Should We Edit Models?cited by 27
  4. 2025The Mirage of Model Editing: Revisiting Evaluation in the Wildcited by 27
  5. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  6. 2024Long-form evaluation of model editingcited by 23
  7. 2023Editing Personality for Large Language Modelscited by 22
  8. 2024Can Editing LLMs Inject Harm?cited by 22
  9. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Modelscited by 20
  10. 2024Representation Shattering in Transformers: A Synthetic Study with Knowledge Editingcited by 19
  11. 2024Agents for self-driving laboratories applied to quantum computingcited by 19
  12. 2024How and where does CLIP process negation?cited by 16
  13. 2023Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Modelscited by 15
  14. 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysiscited by 15
  15. 2023Deceptive Alignment Monitoringcited by 14
  16. 2024Knowledge Localization: Mission Not Accomplished? Enter Query Localization!cited by 13
  17. 2025How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?cited by 13
  18. 2024Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactionscited by 12
  19. 2024Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Modelscited by 12
  20. 2025The Dead Salmons of AI Interpretabilitycited by 12
  21. 2025CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learnerscited by 9
  22. 2025Revealing and Mitigating Over-Attention in Knowledge Editingcited by 8
  23. 2025Context-Robust Knowledge Editing for Language Modelscited by 8
  24. 2026Causality is Key for Interpretability Claims to Generalisecited by 8
  25. 2024TAXI: Evaluating Categorical Knowledge Editing for Language Modelscited by 7
  26. 2025Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injectioncited by 7
  27. 2024Efficiently Quantifying and Mitigating Ripple Effects in Model Editingcited by 6
  28. 2025A Survey on Proactive Defense Strategies Against Misinformation in Large Language Modelscited by 6
  29. 2025The Knowledge Microscope: Features as Better Analytical Lenses than Neuronscited by 5
  30. 2025SEPS: A Separability Measure for Robust Unlearning in LLMscited by 5
  31. 2024How Well Can Knowledge Edit Methods Edit Perplexing Knowledge?cited by 4
  32. 2024Composable Interventions for Language Modelscited by 4
  33. 2025ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphscited by 4
  34. 2024No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Sizecited by 3
  35. 2024SwiftDossier: Tailored Automatic Dossier for Drug Discovery with LLMs and Agentscited by 3
  36. 2024FAME: Towards Factual Multi-Task Model Editingcited by 3
  37. 2026SHIFT: Gate-Modulated Activation Steering for Knowledge Conflict Mitigation in Retrieval-Augmented Generationcited by 3
  38. 2026CLaRE-ty Amid Chaos: Quantifying Representational Entanglement to Predict Ripple Effects in LLM Editingcited by 2

50 most cited of 62 All 62 (JSON)