mlsanityData to 7 Oct 2026

Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models

arXiv 2210.09545 ↗First posted 18 Oct 2022cs.CL cs.CR cs.LGcited by 50

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Deep Neural Networks (DNNs) are known to be vulnerable to backdoor attacks. In Natural Language Processing (NLP), DNNs are often backdoored during the fine-tuning process of a large-scale Pre-trained Language Model (PLM) with poisoned samples. Although the clean weights of PLMs are readily available, existing methods have ignored this information in defending NLP models against backdoor attacks. In this work, we take the first step to exploit the pre-trained (unfine-tuned) weights to mitigate backdoors in fine-tuned language models. Specifically, we leverage the clean pre-trained weights via two complementary techniques: (1) a two-step Fine-mixing technique, which first mixes the backdoored weights (fine-tuned on poisoned data) with the pre-trained weights, then fine-tunes the mixed weights on a small subset of clean data; (2) an Embedding Purification (E-PUR) technique, which mitigates potential backdoors existing in the word embeddings. We compare Fine-mixing with typical backdoor mitigation methods on three single-sentence sentiment classification tasks and two sentence-pair classification tasks and show that it outperforms the baselines by a considerable margin in all scenarios. We also show that our E-PUR method can benefit existing mitigation methods. Our work establishes a simple but strong baseline defense for secure fine-tuned NLP models against backdoor attacks.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 41 of its 58 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

41 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models15 Oct
  2. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May
  3. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models29 Mar

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  12. 2015Generating Sentences from a Continuous Spacecited by 607
26 more
  1. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  2. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 437
  3. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 429
  4. 2020Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networkscited by 341
  5. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  6. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  7. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  8. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  9. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  10. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  11. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  12. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 162
  13. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  14. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  15. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  16. 2022Narcissus: A Practical Clean-Label Backdoor Attack with Limited Informationcited by 102
  17. 2020Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identificationcited by 80
  18. 2018Backdooring Convolutional Neural Networks via Targeted Weight Perturbationscited by 66
  19. 2022Few-Shot Backdoor Attacks on Visual Object Trackingcited by 35
  20. 2020One-pixel Signature: Characterizing CNN Models for Backdoor Detectioncited by 34
  21. 2020Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruptioncited by 26
  22. 2021How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Datacited by 22
  23. 2020Scalable Backdoor Detection in Neural Networkscited by 15
  24. 2020Cassandra: Detecting Trojaned Networks from Adversarial Perturbationscited by 15
  25. 2022Backdoor Attacks on Crowd Countingcited by 9
  26. 2020Noise-Response Analysis of Deep Neural Networks Quantifies Robustness and Fingerprints Structural Malwarecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 19; 2025: 13; 2026: 10.

50 papers.

Sort:

By area:

  1. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  4. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 50
  7. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 39
  8. 2023A Pathway Towards Responsible AI Generated Contentcited by 38
  9. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  10. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  11. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 29
  12. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
38 more
  1. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 26
  2. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 21
  3. 2023Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrationscited by 20
  4. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 19
  5. 2024Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuningcited by 16
  6. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  7. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  8. 2024Simulate and Eliminate: Revoke Backdoors for Generative Large Language Modelscited by 11
  9. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Surveycited by 10
  10. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  11. 2025A Systematic Review of Poisoning Attacks Against Large Language Modelscited by 8
  12. 2023Diffusion Theory as a Scalpel: Detecting and Purifying Poisonous Dimensions in Pre-trained Language Models Caused by Backdoor or Biascited by 7
  13. 2025Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggerscited by 7
  14. 2024Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Spacecited by 6
  15. 2026Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean Referencecited by 6
  16. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  17. 2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Modelscited by 5
  18. 2024Neutralizing Backdoors through Information Conflicts for Large Language Modelscited by 4
  19. 2025Pruning Strategies for Backdoor Defense in LLMscited by 4
  20. 2024Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuitscited by 3
  21. 2025Unmasking Backdoors: An Explainable Defense via Gradient-Attention Anomaly Scoring for Pre-trained Language Modelscited by 3
  22. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  23. 2025NLP Security and Ethics, in the Wildcited by 2
  24. 2025DUP: Detection-guided Unlearning for Backdoor Purification in Language Modelscited by 2
  25. 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Modelscited by 2
  26. 2026Backdooring Bias in Large Language Modelscited by 1
  27. 2026Dummy Backdoor as a Defense: Removing Unknown Backdoors via Shared Internal Mechanisms for Generative LLMscited by 1
  28. 2023SenTest: Evaluating Robustness of Sentence Encoderscited by 0
  29. 2024Cut the Deadwood Out: Backdoor Purification via Guided Module Substitutioncited by 0
  30. 2025Defending against Backdoor Attacks via Module Switchingcited by 0
  31. 2025Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniquescited by 0
  32. 2025Lethe: Purifying Backdoored Large Language Models with Knowledge Dilutioncited by 0
  33. 2026Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspectivecited by 0
  34. 2026Be Kind, Rewrite: Benign Projections via Rewriting Defend Against LLM Data Poisoning Attackscited by 0
  35. 2026LoRAScan: Detecting Backdoor Prompts in Low-Rank Adapters for Large Language Models via Down-Projection Activation Spikescited by 0
  36. 2026Backdoor Decontamination Dynamics in LLM Agentscited by 0
  37. 2026Backdoor Containment via Expert Quarantine and Shutdown in LLMscited by 0
  38. 2026Backdoor Purification for LoRA-Tuned LLMs via Null-Space Projectioncited by 0