# Model Editing at Scale leads to Gradual and Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2401.07453 · HTML: https://mlsanity.com/paper/2401.07453/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 15 Jan 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)

## Built on by (most cited first, 30 of 93)

- [AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models](https://mlsanity.com/paper/2410.02355/) (2024)
- [Detoxifying Large Language Models via Knowledge Editing](https://mlsanity.com/paper/2403.14472/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [A Unified Framework for Model Editing](https://mlsanity.com/paper/2403.14236/) (2024)
- [Larimar: Large Language Models with Episodic Memory Control](https://mlsanity.com/paper/2403.11901/) (2024)
- [On Large Language Model Continual Unlearning](https://mlsanity.com/paper/2407.10223/) (2024)
- [Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing](https://mlsanity.com/paper/2403.07175/) (2024)
- [Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents](https://mlsanity.com/paper/2502.06975/) (2025)
- [Perturbation-Restrained Sequential Model Editing](https://mlsanity.com/paper/2405.16821/) (2024)
- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [The Mirage of Model Editing: Revisiting Evaluation in the Wild](https://mlsanity.com/paper/2502.11177/) (2025)
- [WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs](https://mlsanity.com/paper/2503.05683/) (2025)
- [Avoiding Copyright Infringement via Large Language Model Unlearning](https://mlsanity.com/paper/2406.10952/) (2024)
- [Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models](https://mlsanity.com/paper/2502.10835/) (2025)
- [Can Editing LLMs Inject Harm?](https://mlsanity.com/paper/2407.20224/) (2024)
- [Editing Conceptual Knowledge for Large Language Models](https://mlsanity.com/paper/2403.06259/) (2024)
- [Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering](https://mlsanity.com/paper/2403.19631/) (2024)
- [Neuron-Level Sequential Editing for Large Language Models](https://mlsanity.com/paper/2410.04045/) (2024)
- [Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing](https://mlsanity.com/paper/2410.17194/) (2024)
- [Should We Really Edit Language Models? On the Evaluation of Edited Language Models](https://mlsanity.com/paper/2410.18785/) (2024)
- [BiasEdit: Debiasing Stereotyped Language Models via Model Editing](https://mlsanity.com/paper/2503.08588/) (2025)
- [InstructEdit: Instruction-based Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2402.16123/) (2024)
- [From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries](https://mlsanity.com/paper/2406.12824/) (2024)
- [Position: Editing Large Language Models Poses Serious Safety Risks](https://mlsanity.com/paper/2502.02958/) (2025)
- [Gumbel Counterfactual Generation From Language Models](https://mlsanity.com/paper/2411.07180/) (2024)
- [Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models](https://mlsanity.com/paper/2406.01436/) (2024)
- [Consecutive Batch Model Editing with HooK Layers](https://mlsanity.com/paper/2403.05330/) (2024)
- [Lifelong Knowledge Editing requires Better Regularization](https://mlsanity.com/paper/2502.01636/) (2025)
- [On the Robustness of Editing Large Language Models](https://mlsanity.com/paper/2402.05827/) (2024)
- [MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors](https://mlsanity.com/paper/2405.19086/) (2024)

## Abstract (arXiv)

Editing knowledge in large language models is an attractive capability to have which allows us to correct incorrectly learnt facts during pre-training, as well as update the model with an ever-growing list of new facts. While existing model editing techniques have shown promise, they are usually evaluated using metrics for reliability, specificity and generalization over one or few edits. We argue that for model editing to have practical utility, we must be able to make multiple edits to the same model. With this in mind, we evaluate the current model editing methods at scale, focusing on two state of the art methods: ROME and MEMIT. We find that as the model is edited sequentially with multiple facts, it continually forgets previously edited facts and the ability to perform downstream tasks. This forgetting happens in two phases -- an initial gradual but progressive forgetting phase followed by abrupt or catastrophic forgetting phase. Both gradual and catastrophic forgetting limit the usefulness of model editing methods at scale -- the former making model editing less effective as multiple edits are made to the model while the latter caps the scalability of such model editing methods. Our analysis also highlights other key limitations of ROME and MEMIT at scale. With our work, we push for the development and evaluation of model editing methods keeping scalability in mind.