mlsanityData to 14 Oct 2026

Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks

arXiv 2501.02654 ↗First posted 5 Jan 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent advancements in natural language processing have highlighted the vulnerability of deep learning models to adversarial attacks. While various defence mechanisms have been proposed, there is a lack of comprehensive benchmarks that evaluate these defences across diverse datasets, models, and tasks. In this work, we address this gap by presenting an extensive benchmark for textual adversarial defence that significantly expands upon previous work. Our benchmark incorporates a wide range of datasets, evaluates state-of-the-art defence mechanisms, and extends the assessment to include critical tasks such as single-sentence classification, similarity and paraphrase identification, natural language inference, and commonsense reasoning. This work not only serves as a valuable resource for researchers and practitioners in the field of adversarial robustness but also identifies key areas for future research in textual adversarial defence. By establishing a new standard for benchmarking in this domain, we aim to accelerate progress towards more robust and reliable natural language processing systems.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 48 of its 79 references that are in the map. First posted on arXiv 5 Jan 2025.

Builds on

48 of 79 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2017On Calibration of Modern Neural Networkscited by 4,960
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
36 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  2. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  3. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  4. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  5. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  6. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
  7. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  8. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  9. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
  10. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  11. 2017Generating Natural Adversarial Examplescited by 183
  12. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 163
  13. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  14. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  15. 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 101
  16. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  17. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  18. 2021Towards Improving Adversarial Training of NLP Modelscited by 79
  19. 2021Towards Robustness Against Natural Language Word Substitutionscited by 69
  20. 2020Do We Need Zero Training Loss After Achieving Zero Training Error?cited by 61
  21. 2021Certified Robustness to Text Adversarial Attacks by Randomized [MASK]cited by 57
  22. 2021Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitutioncited by 56
  23. 2020Generating Natural Language Attacks in a Hard Label Black Box Settingcited by 50
  24. 2020Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuningcited by 39
  25. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  26. 2021Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrificecited by 30
  27. 2022Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitutioncited by 28
  28. 2022Query-Efficient and Scalable Black-Box Adversarial Attacks on Discrete Sequential Data via Bayesian Optimizationcited by 22
  29. 2024Advancing the Robustness of Large Language Models through Self-Denoised Smoothingcited by 16
  30. 2023Less is More: Understanding Word-level Textual Adversarial Attack via n-gram Frequency Descendcited by 9
  31. 2024A Modified Word Saliency-Based Adversarial Attack on Text Classification Modelscited by 9
  32. 2024(Chat)GPT v BERT: Dawn of Justice for Semantic Change Detectioncited by 8
  33. 2024Adversarial Text Purification: A Large Language Model Approach for Defensecited by 8
  34. 2022In and Out-of-Domain Text Adversarial Robustness via Label Smoothingcited by 5
  35. 2022Disentangled Text Representation Learning with Information-Theoretic Perspective for Adversarial Robustnesscited by 4
  36. 2024Revisiting Character-level Adversarial Attacks for Language Modelscited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depthcited by 4
  2. 2025Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removalcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models
  2. 2026Unifying Adversarial Robustness and Training Across Text Scoring Models
  3. 2022Rethinking Textual Adversarial Defense for Pre-trained Language Models
  4. 2025Reformulation is All You Need: Addressing Malicious Text Features in DNNs
  5. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
  6. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
  7. 2025Selective Adversarial Attacks on LLM Benchmarks
  8. 2024Robustness of Large Language Models Against Adversarial Attacks