# Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Models

arXiv: https://arxiv.org/abs/2505.23561 · HTML: https://mlsanity.com/paper/2505.23561/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 27 of 27)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger](https://mlsanity.com/paper/2105.12400/) (2021)
- [OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data](https://mlsanity.com/paper/2410.01560/) (2024)
- [Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection](https://mlsanity.com/paper/2307.16888/) (2023)
- [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://mlsanity.com/paper/2312.06795/) (2023)
- [BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents](https://mlsanity.com/paper/2406.03007/) (2024)
- [SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency](https://mlsanity.com/paper/2302.03251/) (2023)
- [Backdoor Pre-trained Models Can Transfer to All](https://mlsanity.com/paper/2111.00197/) (2021)
- [BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models](https://mlsanity.com/paper/2110.02467/) (2021)
- [DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling](https://mlsanity.com/paper/2406.11617/) (2024)
- [BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT](https://mlsanity.com/paper/2304.12298/) (2023)
- [Improving Large Language Model Fine-tuning for Solving Math Problems](https://mlsanity.com/paper/2310.10047/) (2023)
- [BadMerging: Backdoor Attacks Against Model Merging](https://mlsanity.com/paper/2408.07362/) (2024)
- [Here's a Free Lunch: Sanitizing Backdoored Models with Model Merge](https://mlsanity.com/paper/2402.19334/) (2024)
- [Natural Backdoor Attack on Text Data](https://mlsanity.com/paper/2006.16176/) (2020)
- [LoBAM: LoRA-Based Backdoor Attack on Model Merging](https://mlsanity.com/paper/2411.16746/) (2024)
- [Backdoor Attacks to Pre-trained Unified Foundation Models](https://mlsanity.com/paper/2302.09360/) (2023)
- [Physical Backdoor: Towards Temperature-based Backdoor Attacks in the Physical World](https://mlsanity.com/paper/2404.19417/) (2024)

## Built on by (most cited first, 7 of 7)

- [Merge Now, Regret Later: The Hidden Cost of Model Merging Is Adversarial Transferability](https://mlsanity.com/paper/2509.23689/) (2025)
- [Unreal Thinking: Chain-of-Thought Hijacking via Two-stage Backdoor](https://mlsanity.com/paper/2604.09235/) (2026)
- [RogueMerge: Robust and Unified Attacks against LLM Model Merging](https://mlsanity.com/paper/2606.03344/) (2026)
- [From Parameters to Feature Space: Task Arithmetic for Backdoor Mitigation in Model Merging](https://mlsanity.com/paper/2606.12498/) (2026)
- [Delayed Backdoor Attacks: Exploring the Temporal Dimension as a New Attack Surface in Pre-Trained Models](https://mlsanity.com/paper/2603.11949/) (2026)
- [When Privacy Hurts Mergeability: Geometry-Aware Model Merging under Differential Privacy](https://mlsanity.com/paper/2608.26655/) (2026)
- [Towards Merging-conditional Backdoor Unalignment against Large Language Models](https://mlsanity.com/paper/2610.14701/) (2026)

## Abstract (arXiv)

Model merging for Large Language Models (LLMs) directly fuses the parameters of different models finetuned on various tasks, creating a unified model for multi-domain tasks. However, due to potential vulnerabilities in models available on open-source platforms, model merging is susceptible to backdoor attacks. In this paper, we propose Merge Hijacking, the first backdoor attack targeting model merging in LLMs. The attacker constructs a malicious upload model and releases it. Once a victim user merges it with any other models, the resulting merged model inherits the backdoor while maintaining utility across tasks. Merge Hijacking defines two main objectives-effectiveness and utility-and achieves them through four steps. Extensive experiments demonstrate the effectiveness of our attack across different models, merging algorithms, and tasks. Additionally, we show that the attack remains effective even when merging real-world models. Moreover, our attack demonstrates robustness against two inference-time defenses (Paraphrasing and CLEANGEN) and one training-time defense (Fine-pruning).