# Not All Attention Is All You Need

arXiv: https://arxiv.org/abs/2104.04692 · HTML: https://mlsanity.com/paper/2104.04692/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 10 Apr 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Synthesizer: Rethinking Self-Attention in Transformer Models](https://mlsanity.com/paper/2005.00743/) (2020)
- [A* Sampling](https://mlsanity.com/paper/1411.0030/) (2014)
- [SG-Net: Syntax-Guided Machine Reading Comprehension](https://mlsanity.com/paper/1908.05147/) (2019)
- [Hard-Coded Gaussian Attention for Neural Machine Translation](https://mlsanity.com/paper/2005.00742/) (2020)
- [Adversarial Dropout for Supervised and Semi-supervised Learning](https://mlsanity.com/paper/1707.03631/) (2017)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [Code Summarization with Structure-induced Transformer](https://mlsanity.com/paper/2012.14710/) (2020)
- [Scheduled DropHead: A Regularization Method for Transformer Models](https://mlsanity.com/paper/2004.13342/) (2020)
- [Contextual Dropout: An Efficient Sample-Dependent Dropout Module](https://mlsanity.com/paper/2103.04181/) (2021)
- [EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks](https://mlsanity.com/paper/2006.04270/) (2020)
- [How Does Selective Mechanism Improve Self-Attention Networks?](https://mlsanity.com/paper/2005.00979/) (2020)
- [Learnable Bernoulli Dropout for Bayesian Deep Learning](https://mlsanity.com/paper/2002.05155/) (2020)
- [Alleviating the Inequality of Attention Heads for Neural Machine Translation](https://mlsanity.com/paper/2009.09672/) (2020)

## Built on by (most cited first, 5 of 5)

- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection](https://mlsanity.com/paper/2203.14380/) (2022)
- [Adversarial Self-Attention for Language Understanding](https://mlsanity.com/paper/2206.12608/) (2022)
- [Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning](https://mlsanity.com/paper/2210.10293/) (2022)
- [Rethinking Sparse Mixture of Experts from a Unified Perspective](https://mlsanity.com/paper/2503.22996/) (2025)

## Abstract (arXiv)

Beyond the success story of pre-trained language models (PrLMs) in recent natural language processing, they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto self-attention based models, which are broadly chosen as the fundamental architecture of PrLMs. In this paper, we propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning. We demonstrate that state-of-the-art models with elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks.