# Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing

arXiv: https://arxiv.org/abs/2509.01548 · HTML: https://mlsanity.com/paper/2509.01548/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 64 references that are in the map.
- First posted on arXiv 1 Sep 2025.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Equivariant Architectures for Learning in Deep Weight Spaces](https://mlsanity.com/paper/2301.12780/) (2023)
- [Representation Surgery for Multi-Task Model Merging](https://mlsanity.com/paper/2402.02705/) (2024)
- [No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces](https://mlsanity.com/paper/2502.04959/) (2025)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)

## Built on by (most cited first, 6 of 6)

- [Making Models Unmergeable via Scaling-Sensitive Loss Landscape](https://mlsanity.com/paper/2601.21898/) (2026)
- [Generalizing the Geometry of Model Merging Through Frechet Averages](https://mlsanity.com/paper/2604.27155/) (2026)
- [Locking Pretrained Weights via Deep Low-Rank Residual Distillation](https://mlsanity.com/paper/2605.10777/) (2026)
- [On the Vulnerability of Parameter-Level Defenses to Model Merging](https://mlsanity.com/paper/2606.30360/) (2026)
- [Efficient Decentralized Multi-task Dataset Valuation via Model Merging](https://mlsanity.com/paper/2607.03346/) (2026)
- [Retrospective Distillation Attribution via Normalized Response Similarity](https://mlsanity.com/paper/2609.32749/) (2026)

## Abstract (arXiv)

Model merging leverages multiple finetuned expert models to construct a multi-task model with low cost, and is gaining increasing attention. However, as a growing number of finetuned models become publicly available, concerns about the safety of model merging have emerged. Unauthorized merging may infringe on developers' rights and risk leaking sensitive personal information. Most existing methods focus on detecting whether a merged model originates from a specific source model, but fail to effectively prevent illegal merging. In this paper, we propose MergeLock, an active protection mechanism that disrupts model parameters to render them unmergeable, thereby directly preventing unauthorized model merging. Specifically, leveraging the inherent symmetry of the attention mechanism in Transformer-based models, we randomly sample two pairs of invertible matrices and apply them to the Query-Key (QK) and Value-Output (VO) branches. This transformation keeps the model's output unchanged while pushing it away from the shared parameter space of other finetuned models. Extensive experiments across both vision and language tasks demonstrate that MergeLock can degrade the performance of merged models by over 95% when a protected model is involved in most cases, demonstrating its effectiveness. Moreover, we further demonstrate that merged models protected by MergeLock cannot be effectively recovered using low-cost restoration methods, further enhancing robustness against unauthorized merging. The code is available at https://github.com/hetailang/Merge-Lock.