# TCAB: A Large-Scale Text Classification Attack Benchmark

arXiv: https://arxiv.org/abs/2210.12233 · HTML: https://mlsanity.com/paper/2210.12233/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 99 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 30 of 48)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/1909.11764/) (2019)
- [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://mlsanity.com/paper/1911.03437/) (2019)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/1909.00986/) (2019)
- [Combating Adversarial Misspellings with Robust Word Recognition](https://mlsanity.com/paper/1905.11268/) (2019)
- [Contextualized Perturbation for Textual Adversarial Attack](https://mlsanity.com/paper/2009.07502/) (2020)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems](https://mlsanity.com/paper/1903.11508/) (2019)
- [Toxicity Detection: Does Context Really Matter?](https://mlsanity.com/paper/2006.00998/) (2020)
- [InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective](https://mlsanity.com/paper/2010.02329/) (2020)
- [Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation](https://mlsanity.com/paper/1909.01492/) (2019)

## Built on by (most cited first, 2 of 2)

- [CT-GAT: Cross-Task Generative Adversarial Attack based on Transferability](https://mlsanity.com/paper/2310.14265/) (2023)
- [Robust Infidelity: When Faithfulness Measures on Masked Language Models Are Misleading](https://mlsanity.com/paper/2308.06795/) (2023)

## Abstract (arXiv)

We introduce the Text Classification Attack Benchmark (TCAB), a dataset for analyzing, understanding, detecting, and labeling adversarial attacks against text classifiers. TCAB includes 1.5 million attack instances, generated by twelve adversarial attacks targeting three classifiers trained on six source datasets for sentiment analysis and abuse detection in English. Unlike standard text classification, text attacks must be understood in the context of the target classifier that is being attacked, and thus features of the target classifier are important as well. TCAB includes all attack instances that are successful in flipping the predicted label; a subset of the attacks are also labeled by human annotators to determine how frequently the primary semantics are preserved. The process of generating attacks is automated, so that TCAB can easily be extended to incorporate new text attacks and better classifiers as they are developed. In addition to the primary tasks of detecting and labeling attacks, TCAB can also be used for attack localization, attack target labeling, and attack characterization. TCAB code and dataset are available at https://react-nlp.github.io/tcab/.