mlsanityData to 14 Oct 2026

Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrifice

arXiv 2105.14553 ↗First posted 30 May 2021cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Pre-trained contextualized language models (PrLMs) have led to strong performance gains in downstream natural language understanding tasks. However, PrLMs can still be easily fooled by adversarial word substitution, which is one of the most challenging textual adversarial attack methods. Existing defence approaches suffer from notable performance loss and complexities. Thus, this paper presents a compact and performance-preserved framework, Anomaly Detection with Frequency-Aware Randomization (ADFAR). In detail, we design an auxiliary anomaly detection classifier and adopt a multi-task learning procedure, by which PrLMs are able to distinguish adversarial input samples. Then, in order to defend adversarial word substitution, a frequency-aware randomization process is applied to those recognized adversarial input samples. Empirical results show that ADFAR significantly outperforms those newly proposed defense methods over various tasks with much higher inference speed. Remarkably, ADFAR does not impair the overall performance of PrLMs. The code is available at https://github.com/LilyNLP/ADFAR

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 30 May 2021.

Builds on

30 of 46 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Decoupled Weight Decay Regularizationcited by 1,831
  5. 2016Adversarial examples in the physical worldcited by 1,769
  6. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  7. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  8. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  10. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  11. 2018Generating Natural Language Adversarial Examplescited by 424
  12. 2017Mitigating Adversarial Effects Through Randomizationcited by 383
18 more
  1. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
  2. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  3. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  4. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  5. 2019Certified Robustness to Adversarial Word Substitutionscited by 168
  6. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 163
  7. 2016Counter-fitting Word Vectors to Linguistic Constraintscited by 140
  8. 2019Semantics-aware BERT for Language Understandingcited by 112
  9. 2017Deep Text Classification Can be Fooledcited by 106
  10. 2019Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagationcited by 88
  11. 2018On Adversarial Examples for Character-Level Neural Machine Translationcited by 85
  12. 2020SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutionscited by 73
  13. 2019Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classificationcited by 57
  14. 2020Frequency-Guided Word Substitutions for Detecting Textual Adversarial Examplescited by 54
  15. 2019SG-Net: Syntax-Guided Machine Reading Comprehensioncited by 52
  16. 2018Training verified learners with learned verifierscited by 50
  17. 2017The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representationscited by 23
  18. 2017Grammatical Error Correction with Neural Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 6; 2024: 7; 2025: 4; 2026: 2.

30 papers.

Sort:

By area:

  1. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 102
  2. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  3. 2022Detection of Word Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimationcited by 32
  4. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  5. 2022A Survey of Adversarial Defences and Robustness in NLPcited by 23
  6. 2024A Causal Explainable Guardrails for Large Language Modelscited by 18
  7. 2022PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Fillingcited by 9
  8. 2022Adversarial Self-Attention for Language Understandingcited by 8
  9. 2023TextShield: Beyond Successfully Detecting Adversarial Sentences in Text Classificationcited by 8
  10. 2023Don't Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Textcited by 8
  11. 2024Desiderata for the Context Use of Question Answering Systemscited by 7
  12. 2021Improving Question Answering Performance Using Knowledge Distillation and Active Learningcited by 6
18 more
  1. 2022Rethinking Textual Adversarial Defense for Pre-trained Language Modelscited by 5
  2. 2021Adversarial Attacks and Defenses for Social Network Text Processing Applications: Techniques, Challenges and Future Research Directionscited by 4
  3. 2022Disentangled Text Representation Learning with Information-Theoretic Perspective for Adversarial Robustnesscited by 4
  4. 2023The Best Defense is Attack: Repairing Semantics in Textual Adversarial Examplescited by 3
  5. 2024SemRoDe: Macro Adversarial Training to Learn Representations That are Robust to Word-Level Attackscited by 3
  6. 2023On the Universal Adversarial Perturbations for Efficient Data-free Adversarial Detectioncited by 2
  7. 2024Enhance Robustness of Language Models Against Variation Attack through Graph Integrationcited by 2
  8. 2025Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarkscited by 2
  9. 2025NLP Security and Ethics, in the Wildcited by 2
  10. 2025GenoArmory: A Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Modelscited by 2
  11. 2025Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removalcited by 2
  12. 2022RoChBert: Towards Robust BERT Fine-tuning for Chinesecited by 1
  13. 2024DiffuseDef: Improved Robustness to Adversarial Attacks via Iterative Denoisingcited by 1
  14. 2022Using calibrator to improve robustness in Machine Reading Comprehensioncited by 0
  15. 2023SCAT: Robust Self-supervised Contrastive Learning via Adversarial Training for Text Classificationcited by 0
  16. 2023What Learned Representations and Influence Functions Can Tell Us About Adversarial Examplescited by 0
  17. 2026RTD-Guard: A Black-Box Textual Adversarial Detection Framework via Replacement Token Detectioncited by 0
  18. 2026Hybrid Adversarial Defence for Natural Language Understanding Taskscited by 0