# DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

arXiv: https://arxiv.org/abs/1907.11065 · HTML: https://mlsanity.com/paper/1907.11065/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2019.

## Built on by (most cited first, 20 of 20)

- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Causal Transformer for Estimating Counterfactual Outcomes](https://mlsanity.com/paper/2204.07258/) (2022)
- [Latent Neural ODEs with Sparse Bayesian Multiple Shooting](https://mlsanity.com/paper/2210.03466/) (2022)
- [LoRA Meets Dropout under a Unified Framework](https://mlsanity.com/paper/2403.00812/) (2024)
- [Dynamic N:M Fine-grained Structured Sparse Attention Mechanism](https://mlsanity.com/paper/2203.00091/) (2022)
- [M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images](https://mlsanity.com/paper/2409.15092/) (2024)
- [Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers](https://mlsanity.com/paper/2402.04538/) (2024)
- [Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs](https://mlsanity.com/paper/2510.13251/) (2025)
- [Neural Attentive Circuits](https://mlsanity.com/paper/2210.08031/) (2022)
- [Hierarchical Windowed Graph Attention Network and a Large Scale Dataset for Isolated Indian Sign Language Recognition](https://mlsanity.com/paper/2407.14224/) (2024)
- [Language models recognize dropout and Gaussian noise applied to their activations](https://mlsanity.com/paper/2604.17465/) (2026)
- [The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles](https://mlsanity.com/paper/2306.01705/) (2023)
- [DropKey](https://mlsanity.com/paper/2208.02646/) (2022)
- [Reasoning Bias of Next Token Prediction Training](https://mlsanity.com/paper/2502.02007/) (2025)
- [Greener GRASS: Enhancing GNNs with Encoding, Rewiring, and Attention](https://mlsanity.com/paper/2407.05649/) (2024)
- [DropDim: A Regularization Method for Transformer Networks](https://mlsanity.com/paper/2304.10321/) (2023)
- [SAMSA: Efficient Transformer for Many Data Modalities](https://mlsanity.com/paper/2408.05391/) (2024)
- [Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMM](https://mlsanity.com/paper/2410.07531/) (2024)
- [Explicit Dropout: Deterministic Regularization for Transformer Architectures](https://mlsanity.com/paper/2604.20505/) (2026)

## Abstract (arXiv)

Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.