mlsanityData to 7 Oct 2026

Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing

arXiv 2403.07175 ↗First posted 11 Mar 2024cs.CL cs.AIcited by 50

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data

Abstract (arXiv)

Recent work using Rank-One Model Editing (ROME), a popular model editing method, has shown that there are certain facts that the algorithm is unable to edit without breaking the model. Such edits have previously been called disabling edits. These disabling edits cause immediate model collapse and limits the use of ROME for sequential editing. In this paper, we show that disabling edits are an artifact of irregularities in the implementation of ROME. With this paper, we provide a more stable implementation ROME, which we call r-ROME and show that model collapse is no longer observed when making large scale sequential edits with r-ROME, while further improving generalization and locality of model editing compared to the original implementation of ROME. We also provide a detailed mathematical explanation of the reason behind disabling edits.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 16 of its 35 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

16 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse15 Feb

Other references · 15

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2022Mass-Editing Memory in a Transformercited by 976
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2021Editing Factual Knowledge in Language Modelscited by 548
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  7. 2021Fast Model Editing at Scalecited by 491
  8. 2022Memory-Based Model Editing at Scalecited by 443
  9. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  10. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  11. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  12. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
3 more
  1. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  2. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 92
  3. 2024WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editingcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 31; 2026: 9.

50 papers.

Sort:

By area:

  1. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  2. 2024A Unified Framework for Model Editingcited by 54
  3. 2024Perturbation-Restrained Sequential Model Editingcited by 27
  4. 2025WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMscited by 25
  5. 2025The Mirage of Model Editing: Revisiting Evaluation in the Wildcited by 24
  6. 2024Avoiding Copyright Infringement via Large Language Model Unlearningcited by 22
  7. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  8. 2024Memorization in deep learning: A surveycited by 14
  9. 2025Lifelong Knowledge Editing requires Better Regularizationcited by 9
  10. 2024O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editingcited by 8
  11. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  12. 2025UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Modelscited by 7
38 more
  1. 2025Tracing and Reversing Edits in LLMscited by 7
  2. 2025Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMscited by 7
  3. 2025MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modalitycited by 6
  4. 2025Revealing the Deceptiveness of Knowledge Editing: A Mechanistic Analysis of Superficial Editingcited by 5
  5. 2025DocMEdit: Towards Document-Level Model Editingcited by 5
  6. 2025First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimationcited by 5
  7. 2024Enhancing Data Privacy in Large Language Models through Private Association Editingcited by 4
  8. 2024BTBR: A Bayesian-Theory-Driven Probabilistic-Fuzzy Framework for Implicit Bias Removal in Large Language Modelscited by 4
  9. 2024Understanding the Collapse of LLMs in Model Editingcited by 3
  10. 2025LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editingcited by 3
  11. 2025Efficient Knowledge Editing via Minimal Precomputationcited by 3
  12. 2025REPAIR: Robust Editing via Progressive Adaptive Intervention and Reintegrationcited by 3
  13. 2024Keys to Robust Edits: from Theoretical Insights to Practical Advancescited by 2
  14. 2025NAACL2025 Tutorial: Adaptation of Large Language Modelscited by 2
  15. 2025Editing Across Languages: A Survey of Multilingual Knowledge Editingcited by 2
  16. 2025LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Modelscited by 2
  17. 2025InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editingcited by 2
  18. 2025Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinationscited by 2
  19. 2025Model Merging for Knowledge Editingcited by 2
  20. 2025BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responsescited by 2
  21. 2025EvoEdit: Lifelong Free-Text Knowledge Editing through Latent Perturbation Augmentation and Knowledge-driven Parameter Fusioncited by 2
  22. 2025Personality Editing for Language Models through Adjusting Self-Referential Queriescited by 1
  23. 2025Norm Growth and Stability Challenges in Localized Sequential Knowledge Editingcited by 1
  24. 2025Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model Editingcited by 1
  25. 2025QueueEDIT: Structural Self-Correction for Sequential Model Editing in LLMscited by 1
  26. 2025Knowledge Fusion via Bidirectional Information Aggregationcited by 1
  27. 2025STEAM: A Semantic-Level Knowledge Editing Framework for Large Language Modelscited by 1
  28. 2025EtCon: Edit-then-Consolidate for Reliable Knowledge Editingcited by 1
  29. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  30. 2026Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysiscited by 1
  31. 2026MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customizationcited by 1
  32. 2026Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppressioncited by 1
  33. 2025Delta-Audit: Explaining What Changes When Models Changecited by 0
  34. 2026On the Limitations of Rank-One Model Editing in Answering Multi-hop Questionscited by 0
  35. 2026Beyond Local Edits: Embedding-Virtualized Knowledge for Broader Evaluation and Preservation of Model Editingcited by 0
  36. 2026KnowledgeDebugger -- an Exploration Tool for Knowledge Localization and Editing in Transformerscited by 0
  37. 2026Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMscited by 0
  38. 2026Generalizable Lifelong Model Editing via Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing
  2. 2025Rethinking Residual Distribution in Locate-then-Edit Model Editing
  3. 2026The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models
  4. 2025BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing
  5. 2026CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing
  6. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
  7. 2025EMSEdit: Efficient Multi-Step Meta-Learning-based Model Editing
  8. 2025How Robust is Model Editing after Fine-Tuning? An Empirical Study on Text-to-Image Diffusion Models