# Model Merging for Knowledge Editing

arXiv: https://arxiv.org/abs/2506.12384 · HTML: https://mlsanity.com/paper/2506.12384/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 14 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning](https://mlsanity.com/paper/2010.15875/) (2020)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing](https://mlsanity.com/paper/2403.07175/) (2024)
- [LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation](https://mlsanity.com/paper/2405.20646/) (2024)
- [Towards Robust Knowledge Tracing Models via k-Sparse Attention](https://mlsanity.com/paper/2407.17097/) (2024)
- [Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLM](https://mlsanity.com/paper/2312.15450/) (2023)
- [LLMEmb: Large Language Model Can Be a Good Embedding Generator for Sequential Recommendation](https://mlsanity.com/paper/2409.19925/) (2024)

## Built on by (most cited first, 2 of 2)

- [Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging](https://mlsanity.com/paper/2606.18521/) (2026)
- [EtCon: Edit-then-Consolidate for Reliable Knowledge Editing](https://mlsanity.com/paper/2512.04753/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) require continuous updates to maintain accurate and current knowledge as the world evolves. While existing knowledge editing approaches offer various solutions for knowledge updating, they often struggle with sequential editing scenarios and harm the general capabilities of the model, thereby significantly hampering their practical applicability. This paper proposes a two-stage framework combining robust supervised fine-tuning (R-SFT) with model merging for knowledge editing. Our method first fine-tunes the LLM to internalize new knowledge fully, then merges the fine-tuned model with the original foundation model to preserve newly acquired knowledge and general capabilities. Experimental results demonstrate that our approach significantly outperforms existing methods in sequential editing while better preserving the original performance of the model, all without requiring any architectural changes. Code is available at: https://github.com/Applied-Machine-Learning-Lab/MM4KE.