mlsanityData to 7 Oct 2026

Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue

arXiv 2401.04700 ↗First posted 9 Jan 2024cs.CLcited by 101

Not on the map. Why?

Abstract (arXiv)

Model editing is a technique that edits the large language models (LLMs) with updated knowledge to alleviate hallucinations without resource-intensive retraining. While current model editing methods can effectively modify a model's behavior within a specific area of interest, they often overlook the potential unintended side effects on the general abilities of LLMs such as reasoning, natural language inference, and question answering. In this paper, we raise concerns that model editing's improvements on factuality may come at the cost of a significant degradation of the model's general abilities. We systematically analyze the side effects by evaluating four popular editing methods on three LLMs across eight representative tasks. Our extensive empirical experiments show that it is challenging for current editing methods to simultaneously improve factuality of LLMs and maintain their general abilities. Our analysis reveals that the side effects are caused by model editing altering the original model weights excessively, leading to overfitting to the edited facts. To mitigate this, a method named RECT is proposed to regularize the edit update weights by imposing constraints on their complexity based on the RElative Change in weighT. Evaluation results show that RECT can significantly mitigate the side effects of editing while still maintaining over 94% editing performance.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

44 of 56 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
32 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  2. 2022Mass-Editing Memory in a Transformercited by 976
  3. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  4. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  5. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  6. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  7. 2021Editing Factual Knowledge in Language Modelscited by 548
  8. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  9. 2023Erasing Concepts from Diffusion Modelscited by 530
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  11. 2021Fast Model Editing at Scalecited by 491
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  13. 2022Memory-Based Model Editing at Scalecited by 443
  14. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  15. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  16. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  17. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  18. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  19. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  20. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  21. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  22. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  23. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  24. 2020Editable Neural Networkscited by 154
  25. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  26. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 114
  27. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  28. 2023Can We Edit Multimodal Large Language Models?cited by 44
  29. 2023Unveiling the Pitfalls of Knowledge Editing for Large Language Modelscited by 43
  30. 2021Analyzing Commonsense Emergence in Few-shot Knowledge Modelscited by 34
  31. 2023Evaluating the Reversal Curse in Model Editingcited by 34
  32. 2023DocTER: Evaluating Document-based Knowledge Editingcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 47; 2026: 34.

101 papers.

Sort:

By area:

  1. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  2. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  3. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  4. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  5. 2025AnyEdit: Edit Any Knowledge Encoded in Language Modelscited by 74
  6. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  7. 2025Reinforced Lifelong Editing for Language Modelscited by 33
  8. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  9. 2024Perturbation-Restrained Sequential Model Editingcited by 27
  10. 2025The Mirage of Model Editing: Revisiting Evaluation in the Wildcited by 24
  11. 2025PropRAG: Guiding Retrieval with Beam Search over Proposition Pathscited by 22
  12. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Modelscited by 20
38 more
  1. 2024Neuron-Level Sequential Editing for Large Language Modelscited by 18
  2. 2024Representation Shattering in Transformers: A Synthetic Study with Knowledge Editingcited by 18
  3. 2024InstructEdit: Instruction-based Knowledge Editing for Large Language Modelscited by 17
  4. 2024Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questionscited by 15
  5. 2025Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Racecited by 15
  6. 2025Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editingcited by 14
  7. 2024Gumbel Counterfactual Generation From Language Modelscited by 13
  8. 2025CoME: An Unlearning-based Approach to Conflict-free Model Editingcited by 12
  9. 2024Neighboring Perturbations of Knowledge Editing on Large Language Modelscited by 10
  10. 2025Related Knowledge Perturbation Matters: Rethinking Multiple Pieces of Knowledge Editing in Same-Subjectcited by 10
  11. 2025Agents Are All You Need for LLM Unlearningcited by 9
  12. 2025Lifelong Knowledge Editing requires Better Regularizationcited by 9
  13. 2025CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learnerscited by 9
  14. 2025Fine-tuning Done Right in Model Editingcited by 9
  15. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 9
  16. 2024MEMoE: Enhancing Model Editing with Mixture of Experts Adaptorscited by 8
  17. 2024O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editingcited by 8
  18. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  19. 2025Reviving Your MNEME: Predicting The Side Effects of LLM Unlearning and Fine-Tuning via Sparse Model Diffingcited by 8
  20. 2024SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Alteringcited by 7
  21. 2024Identifying Knowledge Editing Types in Large Language Modelscited by 7
  22. 2024Reasons and Solutions for the Decline in Model Performance after Editingcited by 7
  23. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  24. 2025GeoEdit: Geometric Knowledge Editing for Large Language Modelscited by 6
  25. 2025MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modalitycited by 6
  26. 2025COMPKE: Complex Question Answering under Knowledge Editingcited by 6
  27. 2025Is Model Editing Built on Sand? Revealing Its Illusory Success and Fragile Foundationcited by 6
  28. 2024MEGen: Generative Backdoor into Large Language Models via Model Editingcited by 5
  29. 2025Disentangling Knowledge Representations for Large Language Model Editingcited by 5
  30. 2025DocMEdit: Towards Document-Level Model Editingcited by 5
  31. 2024Composable Interventions for Language Modelscited by 4
  32. 2025μKE: Matryoshka Unstructured Knowledge Editing of Large Language Modelscited by 4
  33. 2025ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphscited by 4
  34. 2025Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editingcited by 4
  35. 2025Investigating Model Editing for Unlearning in Large Language Modelscited by 4
  36. 2026From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Modelscited by 4
  37. 2026Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapsecited by 4
  38. 2026Are We Evaluating the Edit Locality of LLM Model Editing Properly?cited by 4

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse
  2. 2025Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoning
  3. 2024Uncovering Overfitting in Large Language Model Editing
  4. 2024FAME: Towards Factual Multi-Task Model Editing
  5. 2026Behemoth: Benchmarking Unlearning in LLMs Using Fully Synthetic Data
  6. 2024Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models
  7. 2023Knowledge Editing for Large Language Models: A Survey
  8. 2025Benchmarking and Rethinking Knowledge Editing for Large Language Models