# Multi-Sample Dropout for Accelerated Training and Better Generalization

arXiv: https://arxiv.org/abs/1905.09788 · HTML: https://mlsanity.com/paper/1905.09788/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2019.

## Built on by (most cited first, 14 of 14)

- [Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction](https://mlsanity.com/paper/2310.11016/) (2023)
- [Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout](https://mlsanity.com/paper/2210.16105/) (2022)
- [On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons](https://mlsanity.com/paper/2112.02668/) (2021)
- [Multi-Task Learning Framework for Extracting Emotion Cause Span and Entailment in Conversations](https://mlsanity.com/paper/2211.03742/) (2022)
- [MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection](https://mlsanity.com/paper/2505.11416/) (2025)
- [Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference](https://mlsanity.com/paper/2601.22132/) (2026)
- [Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction](https://mlsanity.com/paper/2206.00913/) (2022)
- [HAlf-MAsked Model for Named Entity Sentiment analysis](https://mlsanity.com/paper/2308.15793/) (2023)
- [Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning](https://mlsanity.com/paper/2510.10068/) (2025)
- [Predicting Query-Item Relationship using Adversarial Training and Robust Modeling Techniques](https://mlsanity.com/paper/2208.10751/) (2022)
- [Enhancing Model Performance in Multilingual Information Retrieval with Comprehensive Data Engineering Techniques](https://mlsanity.com/paper/2302.07010/) (2023)
- [Rethinking Glaucoma Calibration: Voting-Based Binocular and Metadata Integration](https://mlsanity.com/paper/2503.18642/) (2025)
- [GATech at AbjadGenEval Shared Task: Multilingual Embeddings for Arabic Machine-Generated Text Classification](https://mlsanity.com/paper/2603.10007/) (2026)
- [GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification](https://mlsanity.com/paper/2603.10008/) (2026)

## Abstract (arXiv)

Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, which we call multi-sample dropout, for both accelerating training and improving generalization over the original dropout. The original dropout creates a randomly selected subset (called a dropout sample) from the input in each training iteration while the multi-sample dropout creates multiple dropout samples. The loss is calculated for each sample, and then the sample losses are averaged to obtain the final loss. This technique can be easily implemented by duplicating a part of the network after the dropout layer while sharing the weights among the duplicated fully connected layers. Experimental results using image classification tasks including ImageNet, CIFAR-10, and CIFAR-100 showed that multi-sample dropout accelerates training. Moreover, the networks trained using multi-sample dropout achieved lower error rates compared to networks trained with the original dropout. The additional computation cost due to the duplicated operations is not significant for deep convolutional networks because most of the computation time is consumed in the convolution layers before the dropout layer, which are not duplicated.