# BadMerging: Backdoor Attacks Against Model Merging

arXiv: https://arxiv.org/abs/2408.07362 · HTML: https://mlsanity.com/paper/2408.07362/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 72 references that are in the map.
- First posted on arXiv 14 Aug 2024.

## Builds on (most cited first, 30 of 37)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Attack of the Tails: Yes, You Really Can Backdoor Federated Learning](https://mlsanity.com/paper/2007.05084/) (2020)
- [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019)
- [Anti-Backdoor Learning: Training Clean Models on Poisoned Data](https://mlsanity.com/paper/2110.11571/) (2021)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [A new Backdoor Attack in CNNs by training set corruption without label poisoning](https://mlsanity.com/paper/1902.11237/) (2019)
- [Adversarial Neuron Pruning Purifies Backdoored Deep Models](https://mlsanity.com/paper/2110.14430/) (2021)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [Adversarial Unlearning of Backdoors via Implicit Hypergradient](https://mlsanity.com/paper/2110.03735/) (2021)
- [Backdoor Defense via Decoupling the Training Process](https://mlsanity.com/paper/2202.03423/) (2022)

## Built on by (most cited first, 30 of 34)

- [BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge](https://mlsanity.com/paper/2503.00596/) (2025)
- [Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges](https://mlsanity.com/paper/2409.19993/) (2024)
- [Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace](https://mlsanity.com/paper/2410.13910/) (2024)
- [LoBAM: LoRA-Based Backdoor Attack on Model Merging](https://mlsanity.com/paper/2411.16746/) (2024)
- [Rethinking Weight-Averaged Model-merging](https://mlsanity.com/paper/2411.09263/) (2024)
- [MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Models](https://mlsanity.com/paper/2410.08604/) (2024)
- [Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Models](https://mlsanity.com/paper/2505.23561/) (2025)
- [Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy](https://mlsanity.com/paper/2502.11533/) (2025)
- [Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing](https://mlsanity.com/paper/2410.18267/) (2024)
- [BADTV: Unveiling Backdoor Threats in Third-Party Task Vectors](https://mlsanity.com/paper/2501.02373/) (2025)
- [Causal-Guided Detoxify Backdoor Attack of Open-Weight LoRA Models](https://mlsanity.com/paper/2512.19297/) (2025)
- [LADDER: Multi-objective Backdoor Attack via Evolutionary Algorithm](https://mlsanity.com/paper/2411.19075/) (2024)
- [BadLingual: A Novel Lingual-Backdoor Attack against Large Language Models](https://mlsanity.com/paper/2505.03501/) (2025)
- [LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users](https://mlsanity.com/paper/2507.02850/) (2025)
- [DisarmRAG: Stealthy Retriever-Centric Poisoning to Disable Self-Correction in Retrieval-Augmented Generation (Extended Version)](https://mlsanity.com/paper/2508.20083/) (2025)
- [Merge Now, Regret Later: The Hidden Cost of Model Merging Is Adversarial Transferability](https://mlsanity.com/paper/2509.23689/) (2025)
- [Backdoor Vectors: a Task Arithmetic View on Backdoor Attacks and Defenses](https://mlsanity.com/paper/2510.08016/) (2025)
- [Colluding LoRA: A Compositional Vulnerability in LLM Safety Alignment](https://mlsanity.com/paper/2603.12681/) (2026)
- [FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models](https://mlsanity.com/paper/2604.15488/) (2026)
- [SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation](https://mlsanity.com/paper/2412.19098/) (2024)
- [Backdoor Unlearning by Linear Task Decomposition](https://mlsanity.com/paper/2510.14845/) (2025)
- [RogueMerge: Robust and Unified Attacks against LLM Model Merging](https://mlsanity.com/paper/2606.03344/) (2026)
- [From Parameters to Feature Space: Task Arithmetic for Backdoor Mitigation in Model Merging](https://mlsanity.com/paper/2606.12498/) (2026)
- [Multi-Level Collaboration in Model Merging](https://mlsanity.com/paper/2503.01268/) (2025)
- [BackWeak: Backdooring Knowledge Distillation Simply with Weak Triggers and Fine-tuning](https://mlsanity.com/paper/2511.12046/) (2025)
- [The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training](https://mlsanity.com/paper/2604.07754/) (2026)
- [PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers](https://mlsanity.com/paper/2604.20047/) (2026)
- [Lightweight and Fast Backdoor Model Detection](https://mlsanity.com/paper/2605.18907/) (2026)
- [Stemma: Induced Decision Regions Reveal LLM Provenance](https://mlsanity.com/paper/2607.25880/) (2026)
- [A Security-Oriented Lifecycle Model for Large Language Model Systems](https://mlsanity.com/paper/2608.03626/) (2026)

## Abstract (arXiv)

Fine-tuning pre-trained models for downstream tasks has led to a proliferation of open-sourced task-specific models. Recently, Model Merging (MM) has emerged as an effective approach to facilitate knowledge transfer among these independently fine-tuned models. MM directly combines multiple fine-tuned task-specific models into a merged model without additional training, and the resulting model shows enhanced capabilities in multiple tasks. Although MM provides great utility, it may come with security risks because an adversary can exploit MM to affect multiple downstream tasks. However, the security risks of MM have barely been studied. In this paper, we first find that MM, as a new learning paradigm, introduces unique challenges for existing backdoor attacks due to the merging process. To address these challenges, we introduce BadMerging, the first backdoor attack specifically designed for MM. Notably, BadMerging allows an adversary to compromise the entire merged model by contributing as few as one backdoored task-specific model. BadMerging comprises a two-stage attack mechanism and a novel feature-interpolation-based loss to enhance the robustness of embedded backdoors against the changes of different merging parameters. Considering that a merged model may incorporate tasks from different domains, BadMerging can jointly compromise the tasks provided by the adversary (on-task attack) and other contributors (off-task attack) and solve the corresponding unique challenges with novel attack designs. Extensive experiments show that BadMerging achieves remarkable attacks against various MM algorithms. Our ablation study demonstrates that the proposed attack designs can progressively contribute to the attack performance. Finally, we show that prior defense mechanisms fail to defend against our attacks, highlighting the need for more advanced defense.