mlsanityData to 7 Oct 2026

Adversarial Self-Attention for Language Understanding

arXiv 2206.12608 ↗First posted 25 Jun 2022cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Deep neural models (e.g. Transformer) naturally learn spurious features, which create a ``shortcut'' between the labels and inputs, thus impairing the generalization and robustness. This paper advances the self-attention mechanism to its robust variant for Transformer-based pre-trained language models (e.g. BERT). We propose textitAdversarial Self-Attention mechanism (ASA), which adversarially biases the attentions to effectively suppress the model reliance on features (e.g. specific keywords) and encourage its exploration of broader semantics. We conduct a comprehensive evaluation across a wide range of tasks for both pre-training and fine-tuning stages. For pre-training, ASA unfolds remarkable performance gains compared to naive training for longer steps. For fine-tuning, ASA-empowered models outweigh naive models by a large margin considering both generalization and robustness.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 51 of its 69 references that are in the map. First posted on arXiv 25 Jun 2022.

Builds on

51 of 69 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2020Longformer: The Long-Document Transformercited by 3,362
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  11. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  12. 2019Adversarial Training for Free!cited by 638
  13. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  14. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  15. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  16. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  17. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  18. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  19. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 181
  20. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  21. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  22. 2020Adversarial Training for Large Neural Language Modelscited by 109
  23. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 105
  24. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  25. 2018Interpretable Adversarial Perturbation in Input Embedding Space for Textcited by 67
  26. 2020Robustness to Spurious Correlations via Human Annotationscited by 62
  27. 2019SG-Net: Syntax-Guided Machine Reading Comprehensioncited by 52
  28. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  29. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  30. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  31. 2021HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizationcited by 31
  32. 2021Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrificecited by 30
  33. 2020Code Summarization with Structure-induced Transformercited by 29
  34. 2019A Multi-task Approach for Named Entity Recognition in Social Media Datacited by 18
  35. 2020Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptationcited by 14
  36. 2022Adversarial Training for Improving Model Robustness? Look at Both Prediction and Interpretationcited by 12
  37. 2020Pre-training Text Representations as Meta Learningcited by 5
  38. 2021Not All Attention Is All You Needcited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 1; 2025: 1; 2026: 1.

8 papers.

  1. 2023Chinese Spelling Correction as Rephrasing Language Modelcited by 15
  2. 2023Toward Adversarial Training on Contextualized Language Representationcited by 10
  3. 2023Empower Nested Boolean Logic via Self-Supervised Curriculum Learningcited by 7
  4. 2022Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learningcited by 5
  5. 2023Attack Named Entity Recognition by Entity Boundary Interferencecited by 4
  6. 2025Advances and Challenges in Semantic Textual Similarity: A Comprehensive Surveycited by 1
  7. 2024Unveiling Vulnerability of Self-Attentioncited by 0
  8. 2026Self-Gating Attention for Efficient Time Series Forecastingcited by 0