mlsanityData to 14 Oct 2026

Keys to Robust Edits: from Theoretical Insights to Practical Advances

arXiv 2410.09338 ↗First posted 12 Oct 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) struggle with maintaining accurate knowledge due to conflicting/outdated parametric memories. While locate-and-edit methods address this, their reliance on models' internal representations leads to robustness failures in long-context reasoning and paraphrased queries. We identify a fundamental limitation of locate-and-edit methods: existing semantic keys (for memory localization) cannot simultaneously satisfy robustness (context-invariant activation) and specificity (precise knowledge discrimination). Through theoretical error-bound analysis, we establish formal criteria for effective editing. Our solution introduces textitRobust Edit Pathway (REP), a plug-and-play module that: (1) disentangles editing keys from native model representations; (2) dynamically adjusts keys via contrastive learning to achieve robustness-specificity balance. Extensive experiments across various editing methods (ROME/MEMIT/R-ROME/EMMET), existing LLMs (LLaMA2, QWen, Mistral), and datasets (CounterFact, ZsRE) show that REP improves success rate over robustness tests by up-to 66.4% while maintaining the success rate unaffected. Our code can be found at https://github.com/ElliottYan/RobustKeyEdit .

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 43 of its 48 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

43 of 48 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2022Mass-Editing Memory in a Transformercited by 1,001
  8. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  10. 2021Editing Factual Knowledge in Language Modelscited by 553
  11. 2021Fast Model Editing at Scalecited by 498
  12. 2022Memory-Based Model Editing at Scalecited by 447
31 more
  1. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  2. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  3. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  4. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 197
  5. 2023PMET: Precise Model Editing in a Transformercited by 173
  6. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  7. 2020Editable Neural Networkscited by 155
  8. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 103
  9. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 98
  10. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 93
  11. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  12. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  13. 2023Massive Editing for Large Language Models via Meta Learningcited by 63
  14. 2024The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapsecited by 54
  15. 2024Rebuilding ROME : Resolving Model Collapse during Sequential Model Editingcited by 50
  16. 2023Unveiling the Pitfalls of Knowledge Editing for Large Language Modelscited by 43
  17. 2024WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editingcited by 40
  18. 2024Uncovering Overfitting in Large Language Model Editingcited by 39
  19. 2024Perturbation-Restrained Sequential Model Editingcited by 28
  20. 2024Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Taskscited by 24
  21. 2024Long-form evaluation of model editingcited by 23
  22. 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
  23. 2023Cross-Lingual Knowledge Editing in Large Language Modelscited by 11
  24. 2024Neighboring Perturbations of Knowledge Editing on Large Language Modelscited by 10
  25. 2024Potential and Challenges of Model Editing for Social Debiasingcited by 10
  26. 2024On the Robustness of Editing Large Language Modelscited by 9
  27. 2024Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editingcited by 9
  28. 2023Edit at your own risk: evaluating the robustness of edited models to distribution shiftscited by 8
  29. 2024Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model Editing with Llama-3cited by 8
  30. 2024Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Modelscited by 8
  31. 2024Efficiently Quantifying and Mitigating Ripple Effects in Model Editingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instructioncited by 2
  2. 2026MetaKE: Meta-Learning for Knowledge Editing Toward a Better Accuracy-Editability Trade-offcited by 0
  3. 2026Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidencecited by 0