# A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks

arXiv: https://arxiv.org/abs/2206.08514 · HTML: https://mlsanity.com/paper/2206.08514/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 56 of its 86 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 56)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019)
- [ONION: A Simple and Effective Defense Against Textual Backdoor Attacks](https://mlsanity.com/paper/2011.10369/) (2020)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger](https://mlsanity.com/paper/2105.12400/) (2021)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Detecting AI Trojans Using Meta Neural Analysis](https://mlsanity.com/paper/1910.03137/) (2019)

## Built on by (most cited first, 30 of 74)

- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection](https://mlsanity.com/paper/2307.16888/) (2023)
- [Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP](https://mlsanity.com/paper/2210.10683/) (2022)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023)
- [A Close Look into the Calibration of Pre-trained Language Models](https://mlsanity.com/paper/2211.00151/) (2022)
- [Privacy in Large Language Models: Attacks, Defenses and Future Directions](https://mlsanity.com/paper/2310.10383/) (2023)
- [Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift](https://mlsanity.com/paper/2406.18844/) (2024)
- [Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review](https://mlsanity.com/paper/2309.06055/) (2023)
- [TrojVLM: Backdoor Attack Against Vision Language Models](https://mlsanity.com/paper/2409.19232/) (2024)
- [Backdooring Vision-Language Models with Out-Of-Distribution Data](https://mlsanity.com/paper/2410.01264/) (2024)
- [Attention-Enhancing Backdoor Attacks Against BERT-based Models](https://mlsanity.com/paper/2310.14480/) (2023)
- [Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Sound](https://mlsanity.com/paper/2307.08208/) (2023)
- [From Shortcuts to Triggers: Backdoor Defense with Denoised PoE](https://mlsanity.com/paper/2305.14910/) (2023)
- [Task-Agnostic Detector for Insertion-Based Backdoor Attacks](https://mlsanity.com/paper/2403.17155/) (2024)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [BadActs: A Universal Backdoor Defense in the Activation Space](https://mlsanity.com/paper/2405.11227/) (2024)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLP](https://mlsanity.com/paper/2308.02122/) (2023)
- [Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypots](https://mlsanity.com/paper/2310.18633/) (2023)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2407.19845/) (2024)
- [BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge](https://mlsanity.com/paper/2503.00596/) (2025)
- [TextGuard: Provable Defense against Backdoor Attacks on Text Classification](https://mlsanity.com/paper/2311.11225/) (2023)
- [A Survey on Backdoor Attack and Defense in Natural Language Processing](https://mlsanity.com/paper/2211.11958/) (2022)
- [Large Language Models Are Better Adversaries: Exploring Generative Clean-Label Backdoor Attacks Against Text Classifiers](https://mlsanity.com/paper/2310.18603/) (2023)
- [Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models](https://mlsanity.com/paper/2405.07667/) (2024)
- [Exploring Vulnerabilities and Protections in Large Language Models: A Survey](https://mlsanity.com/paper/2406.00240/) (2024)
- [Demystifying Poisoning Backdoor Attacks from a Statistical Perspective](https://mlsanity.com/paper/2310.10780/) (2023)
- [Rethinking Backdoor Detection Evaluation for Language Models](https://mlsanity.com/paper/2409.00399/) (2024)

## Abstract (arXiv)

Textual backdoor attacks are a kind of practical threat to NLP systems. By injecting a backdoor in the training phase, the adversary could control model predictions via predefined triggers. As various attack and defense models have been proposed, it is of great significance to perform rigorous evaluations. However, we highlight two issues in previous backdoor learning evaluations: (1) The differences between real-world scenarios (e.g. releasing poisoned datasets or models) are neglected, and we argue that each scenario has its own constraints and concerns, thus requires specific evaluation protocols; (2) The evaluation metrics only consider whether the attacks could flip the models' predictions on poisoned samples and retain performances on benign samples, but ignore that poisoned samples should also be stealthy and semantic-preserving. To address these issues, we categorize existing works into three practical scenarios in which attackers release datasets, pre-trained models, and fine-tuned models respectively, then discuss their unique evaluation methodologies. On metrics, to completely evaluate poisoned samples, we use grammar error increase and perplexity difference for stealthiness, along with text similarity for validity. After formalizing the frameworks, we develop an open-source toolkit OpenBackdoor to foster the implementations and evaluations of textual backdoor learning. With this toolkit, we perform extensive experiments to benchmark attack and defense models under the suggested paradigm. To facilitate the underexplored defenses against poisoned datasets, we further propose CUBE, a simple yet strong clustering-based defense baseline. We hope that our frameworks and benchmarks could serve as the cornerstones for future model development and evaluations.