# Understanding the Collapse of LLMs in Model Editing

arXiv: https://arxiv.org/abs/2406.11263 · HTML: https://mlsanity.com/paper/2406.11263/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Model Editing at Scale leads to Gradual and Catastrophic Forgetting](https://mlsanity.com/paper/2401.07453/) (2024)
- [The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse](https://mlsanity.com/paper/2402.09656/) (2024)
- [Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing](https://mlsanity.com/paper/2403.07175/) (2024)

## Built on by (most cited first, 3 of 3)

- [Related Knowledge Perturbation Matters: Rethinking Multiple Pieces of Knowledge Editing in Same-Subject](https://mlsanity.com/paper/2502.06868/) (2025)
- [Mitigating Gender Bias in Code Large Language Models via Model Editing](https://mlsanity.com/paper/2410.07820/) (2024)
- [Circuit-Diff: Factual Edit-based Intervention Method for Localizing Knowledge in Attribution Graphs](https://mlsanity.com/paper/2609.23892/) (2026)

## Abstract (arXiv)

Despite significant progress in model editing methods, their application in real-world scenarios remains challenging as they often cause large language models (LLMs) to collapse. Among them, ROME is particularly concerning, as it could disrupt LLMs with only a single edit. In this paper, we study the root causes of such collapse. Through extensive analysis, we identify two primary factors that contribute to the collapse: i) inconsistent handling of prefixed and unprefixed keys in the parameter update equation may result in very small denominators, causing excessively large parameter updates; ii) the subject of collapse cases is usually the first token, whose unprefixed key distribution significantly differs from the prefixed key distribution in autoregressive transformers, causing the aforementioned issue to materialize. To validate our findings, we propose a simple yet effective approach: uniformly using prefixed keys during editing phase and adding prefixes during testing phase to ensure the consistency between training and testing. The experimental results show that the proposed solution can prevent model collapse while maintaining the effectiveness of the edits.