mlsanityData to 14 Oct 2026

DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

arXiv 1907.11065 ↗First posted 25 Jul 2019cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 3; 2024: 7; 2025: 2; 2026: 2.

20 papers.

Sort:

By area:

  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  2. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  3. 2022Causal Transformer for Estimating Counterfactual Outcomescited by 123
  4. 2022Latent Neural ODEs with Sparse Bayesian Multiple Shootingcited by 17
  5. 2024LoRA Meets Dropout under a Unified Frameworkcited by 17
  6. 2022Dynamic N:M Fine-grained Structured Sparse Attention Mechanismcited by 14
  7. 2024M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Imagescited by 13
  8. 2024Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformerscited by 12
  9. 2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 10
  10. 2022Neural Attentive Circuitscited by 6
  11. 2024Hierarchical Windowed Graph Attention Network and a Large Scale Dataset for Isolated Indian Sign Language Recognitioncited by 6
  12. 2026Language models recognize dropout and Gaussian noise applied to their activationscited by 5
8 more
  1. 2023The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensemblescited by 4
  2. 2022DropKeycited by 3
  3. 2025Reasoning Bias of Next Token Prediction Trainingcited by 3
  4. 2024Greener GRASS: Enhancing GNNs with Encoding, Rewiring, and Attentioncited by 2
  5. 2023DropDim: A Regularization Method for Transformer Networkscited by 1
  6. 2024SAMSA: Efficient Transformer for Many Data Modalitiescited by 0
  7. 2024Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMMcited by 0
  8. 2026Explicit Dropout: Deterministic Regularization for Transformer Architecturescited by 0