# Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks

arXiv: https://arxiv.org/abs/2101.06969 · HTML: https://mlsanity.com/paper/2101.06969/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 67 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks](https://mlsanity.com/paper/2101.05930/) (2021)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Adversarial Attacks and Defenses in Images, Graphs and Text: A Review](https://mlsanity.com/paper/1909.08072/) (2019)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [Detecting AI Trojans Using Meta Neural Analysis](https://mlsanity.com/paper/1910.03137/) (2019)
- [Blind Backdoors in Deep Learning Models](https://mlsanity.com/paper/2005.03823/) (2020)
- [Trojaning Language Models for Fun and Profit](https://mlsanity.com/paper/2008.00312/) (2020)

## Built on by (most cited first, 30 of 49)

- [Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning](https://mlsanity.com/paper/2108.13888/) (2021)
- [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023)
- [Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures](https://mlsanity.com/paper/2112.05224/) (2021)
- [Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark](https://mlsanity.com/paper/2305.10036/) (2023)
- [BadPrompt: Backdoor Attacks on Continuous Prompts](https://mlsanity.com/paper/2211.14719/) (2022)
- [Exploring the Universal Vulnerability of Prompt-based Learning Paradigm](https://mlsanity.com/paper/2204.05239/) (2022)
- [BITE: Textual Backdoor Attacks with Iterative Trigger Injection](https://mlsanity.com/paper/2205.12700/) (2022)
- [Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defense](https://mlsanity.com/paper/2202.05749/) (2022)
- [UNICORN: A Unified Backdoor Trigger Inversion Framework](https://mlsanity.com/paper/2304.02786/) (2023)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review](https://mlsanity.com/paper/2309.06055/) (2023)
- [Pre-trained Trojan Attacks for Visual Recognition](https://mlsanity.com/paper/2312.15172/) (2023)
- [NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models](https://mlsanity.com/paper/2305.17826/) (2023)
- [BAGM: A Backdoor Attack for Manipulating Text-to-Image Generative Models](https://mlsanity.com/paper/2307.16489/) (2023)
- [Attention-Enhancing Backdoor Attacks Against BERT-based Models](https://mlsanity.com/paper/2310.14480/) (2023)
- [Training-free Lexical Backdoor Attacks on Language Models](https://mlsanity.com/paper/2302.04116/) (2023)
- [Watermarking Graph Neural Networks based on Backdoor Attacks](https://mlsanity.com/paper/2110.11024/) (2021)
- [Grounding Foundation Models through Federated Transfer Learning: A General Framework](https://mlsanity.com/paper/2311.17431/) (2023)
- [Multi-target Backdoor Attacks for Code Pre-trained Models](https://mlsanity.com/paper/2306.08350/) (2023)
- [Watermarking LLMs with Weight Quantization](https://mlsanity.com/paper/2310.11237/) (2023)
- [Double-I Watermark: Protecting Model Copyright for LLM Fine-tuning](https://mlsanity.com/paper/2402.14883/) (2024)
- [ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLP](https://mlsanity.com/paper/2308.02122/) (2023)
- [TextGuard: Provable Defense against Backdoor Attacks on Text Classification](https://mlsanity.com/paper/2311.11225/) (2023)
- [A Survey on Backdoor Attack and Defense in Natural Language Processing](https://mlsanity.com/paper/2211.11958/) (2022)
- [WARDEN: Multi-Directional Backdoor Watermarks for Embedding-as-a-Service Copyright Protection](https://mlsanity.com/paper/2403.01472/) (2024)
- [LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors](https://mlsanity.com/paper/2308.13904/) (2023)
- [Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm](https://mlsanity.com/paper/2409.14119/) (2024)
- [UOR: Universal Backdoor Attacks on Pre-trained Language Models](https://mlsanity.com/paper/2305.09574/) (2023)
- [Mutual Information Guided Backdoor Mitigation for Pre-trained Encoders](https://mlsanity.com/paper/2406.03508/) (2024)
- [MEGen: Generative Backdoor into Large Language Models via Model Editing](https://mlsanity.com/paper/2408.10722/) (2024)

## Abstract (arXiv)

Pre-trained models (PTMs) have been widely used in various downstream tasks. The parameters of PTMs are distributed on the Internet and may suffer backdoor attacks. In this work, we demonstrate the universal vulnerability of PTMs, where fine-tuned PTMs can be easily controlled by backdoor attacks in arbitrary downstream tasks. Specifically, attackers can add a simple pre-training task, which restricts the output representations of trigger instances to pre-defined vectors, namely neuron-level backdoor attack (NeuBA). If the backdoor functionality is not eliminated during fine-tuning, the triggers can make the fine-tuned model predict fixed labels by pre-defined vectors. In the experiments of both natural language processing (NLP) and computer vision (CV), we show that NeuBA absolutely controls the predictions for trigger instances without any knowledge of downstream tasks. Finally, we apply several defense methods to NeuBA and find that model pruning is a promising direction to resist NeuBA by excluding backdoored neurons. Our findings sound a red alarm for the wide use of PTMs. Our source code and models are available at urlhttps://github.com/thunlp/NeuBA.