mlsanityData to 7 Oct 2026

Improved Text Classification via Contrastive Adversarial Training

arXiv 2107.10137 ↗First posted 21 Jul 2021cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

We propose a simple and general method to regularize the fine-tuning of Transformer-based encoders for text classification tasks. Specifically, during fine-tuning we generate adversarial examples by perturbing the word embeddings of the model and perform contrastive learning on clean and adversarial examples in order to teach the model to learn noise-invariant representations. By training on both clean and adversarial examples along with the additional contrastive objective, we observe consistent improvement over standard fine-tuning on clean examples. On several GLUE benchmark tasks, our fine-tuned BERT Large model outperforms BERT Large baseline by 1.7% on average, and our fine-tuned RoBERTa Large improves over RoBERTa Large baseline by 1.3%. We additionally validate our method in different domains using three intent classification datasets, where our fine-tuned RoBERTa Large outperforms RoBERTa Large baseline by 1-2% on average.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 46 of its 51 references that are in the map. First posted on arXiv 21 Jul 2021.

Builds on

46 of 51 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2020Supervised Contrastive Learningcited by 2,803
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
34 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  6. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  7. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  8. 2019Adversarial Training for Free!cited by 638
  9. 2017One pixel attack for fooling deep neural networkscited by 579
  10. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  11. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  12. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  13. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  14. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  15. 2017Adversarial Examples for Semantic Segmentation and Object Detectioncited by 275
  16. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  17. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  18. 2018Adversarial Attacks and Defences: A Surveycited by 259
  19. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  20. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  21. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  22. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  23. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  24. 2018Physical Adversarial Examples for Object Detectorscited by 163
  25. 2018ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detectorcited by 159
  26. 2017On the Robustness of Semantic Segmentation Models to Adversarial Attackscited by 126
  27. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  28. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  29. 2015Distributional Smoothing with Virtual Adversarial Trainingcited by 61
  30. 2020On Learning Universal Representations Across Languagescited by 58
  31. 2020Multilingual BERT Post-Pretraining Alignmentcited by 30
  32. 2020Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Trainingcited by 12
  33. 2020Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluationscited by 8
  34. 2021Making Attention Mechanisms More Robust and Interpretable with Virtual Adversarial Trainingcited by 8

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 10; 2023: 9; 2024: 14; 2025: 4.

38 papers.

Sort:

By area:

  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  2. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  3. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  4. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  5. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  6. 2022Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classificationcited by 52
  7. 2023ConDA: Contrastive Domain Adaptation for AI-generated Text Detectioncited by 37
  8. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  9. 2021Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillationcited by 22
  10. 2023Towards Viewpoint-Invariant Visual Recognition via Adversarial Trainingcited by 19
  11. 2024Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgerycited by 17
  12. 2024Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationalescited by 16
26 more
  1. 2022Supervised Contrastive Learning with Hard Negative Samplescited by 13
  2. 2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Modelscited by 12
  3. 2024Is Contrasting All You Need? Contrastive Learning for the Detection and Attribution of AI-generated Textcited by 11
  4. 2022Impact of Adversarial Training on Robustness and Generalizability of Language Modelscited by 9
  5. 2023HateProof: Are Hateful Meme Detection Systems really Robust?cited by 7
  6. 2024Adversarial Training: A Surveycited by 6
  7. 2022Cyberbullying Classifiers are Sensitive to Model-Agnostic Perturbationscited by 5
  8. 2023Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-trainingcited by 5
  9. 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignmentcited by 5
  10. 2023VendorLink: An NLP approach for Identifying & Linking Vendor Migrants & Potential Aliases on Darknet Marketscited by 3
  11. 2024Vision HgNN: An Electron-Micrograph is Worth Hypergraph of Hypernodescited by 3
  12. 2022Two-Level Supervised Contrastive Learning for Response Selection in Multi-Turn Dialoguecited by 2
  13. 2022A Novel Approach to Train Diverse Types of Language Models for Health Mention Classification of Tweetscited by 2
  14. 2022A Benchmark Study of Contrastive Learning for Arabic Social Meaningcited by 2
  15. 2023Proxy-based Zero-Shot Entity Linking by Effective Candidate Retrievalcited by 2
  16. 2024RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learningcited by 2
  17. 2024Multi-label Sequential Sentence Classification via Large Language Modelcited by 2
  18. 2025Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarkscited by 2
  19. 2025SECRET: Semi-supervised Clinical Trial Document Similarity Searchcited by 2
  20. 2022Improving Health Mentioning Classification of Tweets using Contrastive Adversarial Trainingcited by 1
  21. 2022Contrastive Learning of Sociopragmatic Meaning in Social Mediacited by 1
  22. 2022Zero-shot stance detection based on cross-domain feature enhancement by contrastive learningcited by 1
  23. 2023DEFTri: A Few-Shot Label Fused Contextual Representation Learning For Product Defect Triage in e-Commercecited by 0
  24. 2024QLSC: A Query Latent Semantic Calibrator for Robust Extractive Question Answeringcited by 0
  25. 2025A Simple Graph Contrastive Learning Framework for Short Text Classificationcited by 0
  26. 2025Indirect Attention: Turning Context Misalignment into a Featurecited by 0