mlsanityData to 13 Oct 2026

Not All Attention Is All You Need

arXiv 2104.04692 ↗First posted 10 Apr 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Beyond the success story of pre-trained language models (PrLMs) in recent natural language processing, they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto self-attention based models, which are broadly chosen as the fundamental architecture of PrLMs. In this paper, we propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning. We demonstrate that state-of-the-art models with elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 10 Apr 2021.

Builds on

29 of 40 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,171
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,789
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,496
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,469
  10. 2017Self-Normalizing Neural Networkscited by 594
  11. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 436
17 more
  1. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 413
  2. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  3. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  4. 2017Concrete Dropoutcited by 227
  5. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  6. 2014A* Samplingcited by 150
  7. 2019SG-Net: Syntax-Guided Machine Reading Comprehensioncited by 52
  8. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  9. 2017Adversarial Dropout for Supervised and Semi-supervised Learningcited by 38
  10. 2021AutoDropout: Learning Dropout Patterns to Regularize Deep Networkscited by 35
  11. 2020Code Summarization with Structure-induced Transformercited by 29
  12. 2020Scheduled DropHead: A Regularization Method for Transformer Modelscited by 28
  13. 2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
  14. 2020EDropout: Energy-Based Dropout and Pruning of Deep Neural Networkscited by 20
  15. 2020How Does Selective Mechanism Improve Self-Attention Networks?cited by 18
  16. 2020Learnable Bernoulli Dropout for Bayesian Deep Learningcited by 14
  17. 2020Alleviating the Inequality of Attention Heads for Neural Machine Translationcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 0; 2024: 0; 2025: 1.

5 papers.

  1. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  2. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  3. 2022Adversarial Self-Attention for Language Understandingcited by 8
  4. 2022Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learningcited by 5
  5. 2025Rethinking Sparse Mixture of Experts from a Unified Perspectivecited by 1