TCAB: A Large-Scale Text Classification Attack Benchmark
Not on the map. Why?
Abstract (arXiv)
We introduce the Text Classification Attack Benchmark (TCAB), a dataset for analyzing, understanding, detecting, and labeling adversarial attacks against text classifiers. TCAB includes 1.5 million attack instances, generated by twelve adversarial attacks targeting three classifiers trained on six source datasets for sentiment analysis and abuse detection in English. Unlike standard text classification, text attacks must be understood in the context of the target classifier that is being attacked, and thus features of the target classifier are important as well. TCAB includes all attack instances that are successful in flipping the predicted label; a subset of the attacks are also labeled by human annotators to determine how frequently the primary semantics are preserved. The process of generating attacks is automated, so that TCAB can easily be extended to incorporate new text attacks and better classifiers as they are developed. In addition to the primary tasks of detecting and labeling attacks, TCAB can also be used for attack localization, attack target labeling, and attack characterization. TCAB code and dataset are available at https://react-nlp.github.io/tcab/.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 48 of its 99 references that are in the map. First posted on arXiv 21 Oct 2022.
Builds on
48 of 99 references.
References · 48
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2018Datasheets for Datasetscited by 1,371
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
- 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
- 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
- 2018Universal Sentence Encodercited by 485
- 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
36 more
- 2018Generating Natural Language Adversarial Examplescited by 422
- 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
- 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
- 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
- 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
- 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
- 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
- 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
- 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
- 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
- 2019Certified Robustness to Adversarial Word Substitutionscited by 164
- 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
- 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
- 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
- 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 99
- 2020Toxicity Detection: Does Context Really Matter?cited by 93
- 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
- 2019Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagationcited by 87
- 2020Contextualizing Hate Speech Classifiers with Post-hoc Explanationcited by 84
- 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
- 2020Reevaluating Adversarial Examples in Natural Languagecited by 83
- 2021Towards Improving Adversarial Training of NLP Modelscited by 79
- 2020Robustness Verification for Transformerscited by 76
- 2021Towards Robustness Against Natural Language Word Substitutionscited by 68
- 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
- 2020Robust Encodings: A Framework for Combating Adversarial Typoscited by 58
- 2019Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classificationcited by 55
- 2020Frequency-Guided Word Substitutions for Detecting Textual Adversarial Examplescited by 54
- 2020Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weightingcited by 41
- 2019Word-level Textual Adversarial Attacking as Combinatorial Optimizationcited by 37
- 2021Exploring the Efficacy of Automatically Generated Counterfactuals for Sentiment Analysiscited by 30
- 2021Achieving Model Robustness through Discrete Adversarial Trainingcited by 23
- 2020Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encodingcited by 22
- 2022Effective Token Graph Modeling using a Novel Labeling Strategy for Structured Sentiment Analysiscited by 22
- 2022Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysiscited by 21
- 2020A Sweet Rabbit Hole by DARCY: Using Honeypots to Detect Universal Trigger's Adversarial Attackscited by 14
Built on by
Citing papers per year: 2023: 2.
2 papers.