mlsanityData to 7 Oct 2026

Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models

arXiv 2601.04448 ↗First posted 7 Jan 2026cs.CL cs.AIcited by 2

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Large Language Models (LLMs) have greatly advanced Natural Language Processing (NLP), particularly through instruction tuning, which enables broad task generalization without additional fine-tuning. However, their reliance on large-scale datasets-often collected from human or web sources-makes them vulnerable to backdoor attacks, where adversaries poison a small subset of data to implant hidden behaviors. Despite this growing risk, defenses for instruction-tuned models remain underexplored. We propose MB-Defense (Merging & Breaking Defense Framework), a novel training pipeline that immunizes instruction-tuned LLMs against diverse backdoor threats. MB-Defense comprises two stages: (i) Defensive Poisoning, which merges attacker and defensive triggers into a unified backdoor representation, and (ii) Backdoor Neutralization, which breaks this representation through additional training to restore clean behavior. Extensive experiments across multiple LLMs show that MB-Defense substantially lowers attack success rates while preserving instruction-following ability. Our method offers a generalizable and data-efficient defense strategy, improving the robustness of instruction-tuned LLMs against unseen backdoor attacks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 40 of its 63 references that are in the map. First posted on arXiv 7 Jan 2026.

Builds on

40 of 63 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks23 Sep
  2. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection31 Jul
  3. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models2 May
  4. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger27 Apr
  5. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models18 Oct
  6. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injection25 May
  7. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer14 Oct
  8. 2021Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution11 Jun
  9. 2021Defending Against Backdoor Attacks in Natural Language Generation3 Jun
  10. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2017Decoupled Weight Decay Regularizationcited by 1,722
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2023LIMA: Less Is More for Alignmentcited by 1,080
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  10. 2023Whose Opinions Do Language Models Reflect?cited by 759
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  12. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
18 more
  1. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  2. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  4. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  5. 2023Poisoning Language Models During Instruction Tuningcited by 254
  6. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  7. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  8. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  9. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  10. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  11. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agentscited by 106
  12. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  13. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  14. 2022A Study of the Attention Abnormality in Trojaned BERTscited by 47
  15. 2022CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learningcited by 32
  16. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 26
  17. 2023Quantifying the Impact of Large Language Models on Collective Opinion Dynamicscited by 22
  18. 2024Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agentscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Backdoor Decontamination Dynamics in LLM Agentscited by 0
  2. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0