# Enhancing Transformer Training Efficiency with Dynamic Dropout

arXiv: https://arxiv.org/abs/2411.03236 · HTML: https://mlsanity.com/paper/2411.03236/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 5 Nov 2024.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning to Represent Patches](https://mlsanity.com/paper/2308.16586/) (2023)
- [Domain Adaptive Graph Classification](https://mlsanity.com/paper/2312.13536/) (2023)

## Built on by (most cited first, 1 of 1)

- [Training-Efficient Text-to-Music Generation with State-Space Modeling](https://mlsanity.com/paper/2601.14786/) (2026)

## Abstract (arXiv)

We introduce Dynamic Dropout, a novel regularization technique designed to enhance the training efficiency of Transformer models by dynamically adjusting the dropout rate based on training epochs or validation loss improvements. This approach addresses the challenge of balancing regularization and model capacity, which is crucial for achieving fast convergence and high performance. Our method involves modifying the GPT model to accept a variable dropout rate and updating dropout layers during training using schedules such as linear decay, exponential decay, and validation loss-based adjustments. Extensive experiments on the Shakespearechar dataset demonstrate that Dynamic Dropout significantly accelerates training and improves inference efficiency compared to a baseline model with a fixed dropout rate. The validation loss-based adjustment schedule provided the best overall performance, highlighting the potential of Dynamic Dropout as a valuable technique for training large-scale Transformer models.