mlsanityData to 7 Oct 2026

Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks

arXiv 2107.07610 ↗First posted 15 Jul 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we present an approach to improve the robustness of BERT language models against word substitution-based adversarial attacks by leveraging adversarial perturbations for self-supervised contrastive learning. We create a word-level adversarial attack generating hard positives on-the-fly as adversarial examples during contrastive learning. In contrast to previous works, our method improves model robustness without using any labeled data. Experimental results show that our method improves robustness of BERT against four different word substitution-based adversarial attacks, and combining our method with adversarial training gives higher robustness than adversarial training alone. As our method improves the robustness of BERT purely with unlabeled data, it opens up the possibility of using large text datasets to train robust language models against word substitution-based adversarial attacks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 15 Jul 2021.

Builds on

22 of 30 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  8. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  9. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  10. 2018Generating Natural Language Adversarial Examplescited by 422
  11. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  12. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
10 more
  1. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  2. 2016Counter-fitting Word Vectors to Linguistic Constraintscited by 138
  3. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  4. 2020Adversarial Training for Large Neural Language Modelscited by 109
  5. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  6. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  7. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  8. 2020Contrastive Learning with Adversarial Perturbations for Conditional Text Generationcited by 56
  9. 2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
  10. 2020A Geometry-Inspired Attack for Generating Natural Language Adversarial Examplescited by 20

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  2. 2024Assessing Adversarial Robustness of Large Language Models: An Empirical Studycited by 21
  3. 2023Learning under Label Proportions for Text Classificationcited by 1
  4. 2025Unsupervised Candidate Ranking for Lexical Substitution via Holistic Sentence Semanticscited by 0