# Scheduled DropHead: A Regularization Method for Transformer Models

arXiv: https://arxiv.org/abs/2004.13342 · HTML: https://mlsanity.com/paper/2004.13342/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2020.

## Built on by (most cited first, 28 of 28)

- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization](https://mlsanity.com/paper/2105.12002/) (2021)
- [Long-Range Transformers for Dynamic Spatiotemporal Forecasting](https://mlsanity.com/paper/2109.12218/) (2021)
- [Faster Causal Attention Over Large Sequences Through Sparse Flash Attention](https://mlsanity.com/paper/2306.01160/) (2023)
- [Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations](https://mlsanity.com/paper/2109.13059/) (2021)
- [MHAttnSurv: Multi-Head Attention for Survival Prediction Using Whole-Slide Pathology Images](https://mlsanity.com/paper/2110.11558/) (2021)
- [Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling](https://mlsanity.com/paper/2110.03252/) (2021)
- [Dynamic Masking Rate Schedules for MLM Pretraining](https://mlsanity.com/paper/2305.15096/) (2023)
- [UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost](https://mlsanity.com/paper/2104.04946/) (2021)
- [How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?](https://mlsanity.com/paper/2109.01100/) (2021)
- [On Biasing Transformer Attention Towards Monotonicity](https://mlsanity.com/paper/2104.03945/) (2021)
- [Not All Attention Is All You Need](https://mlsanity.com/paper/2104.04692/) (2021)
- [Relaxed Attention for Transformer Models](https://mlsanity.com/paper/2209.09735/) (2022)
- [Language models recognize dropout and Gaussian noise applied to their activations](https://mlsanity.com/paper/2604.17465/) (2026)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles](https://mlsanity.com/paper/2306.01705/) (2023)
- [On Separate Normalization in Self-supervised Transformers](https://mlsanity.com/paper/2309.12931/) (2023)
- [Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods](https://mlsanity.com/paper/2510.05901/) (2025)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [DropDim: A Regularization Method for Transformer Networks](https://mlsanity.com/paper/2304.10321/) (2023)
- [Multi-Objective Decision Transformers for Offline Reinforcement Learning](https://mlsanity.com/paper/2308.16379/) (2023)
- [TLM: Token-Level Masking for Transformers](https://mlsanity.com/paper/2310.18738/) (2023)
- [Regularized Contrastive Learning of Semantic Search](https://mlsanity.com/paper/2209.13241/) (2022)
- [Semi-Structured Object Sequence Encoders](https://mlsanity.com/paper/2301.01015/) (2023)
- [AttentionDrop: A Novel Regularization Method for Transformer Models](https://mlsanity.com/paper/2504.12088/) (2025)
- [Massive Redundancy in Gradient Transport Enables Sparse Online Learning](https://mlsanity.com/paper/2603.15195/) (2026)
- [Explicit Dropout: Deterministic Regularization for Transformer Architectures](https://mlsanity.com/paper/2604.20505/) (2026)

## Abstract (arXiv)

In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various NLP tasks. In contrast to the conventional dropout mechanisms which randomly drop units or connections, the proposed DropHead is a structured dropout method. It drops entire attention-heads during training and It prevents the multi-head attention model from being dominated by a small portion of attention heads while also reduces the risk of overfitting the training data, thus making use of the multi-head attention mechanism more efficiently. Motivated by recent studies about the learning dynamic of the multi-head attention mechanism, we propose a specific dropout rate schedule to adaptively adjust the dropout rate of DropHead and achieve better regularization effect. Experimental results on both machine translation and text classification benchmark datasets demonstrate the effectiveness of the proposed approach.