# Dropout with Expectation-linear Regularization

arXiv: https://arxiv.org/abs/1609.08017 · HTML: https://mlsanity.com/paper/1609.08017/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2016.

## Built on by (most cited first, 9 of 9)

- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding](https://mlsanity.com/paper/2301.03765/) (2023)
- [Improving Image Captioning with Control Signal of Sentence Quality](https://mlsanity.com/paper/2206.03196/) (2022)
- [R-Block: Regularized Block of Dropout for convolutional networks](https://mlsanity.com/paper/2307.15150/) (2023)
- [CR-SFP: Learning Consistent Representation for Soft Filter Pruning](https://mlsanity.com/paper/2312.11555/) (2023)
- [Invariance-powered Trustworthy Defense via Remove Then Restore](https://mlsanity.com/paper/2402.00304/) (2024)
- [Layer-wise Regularized Dropout for Neural Language Models](https://mlsanity.com/paper/2402.16361/) (2024)

## Abstract (arXiv)

Dropout, a simple and effective way to train deep neural networks, has led to a number of impressive empirical successes and spawned many recent theoretical investigations. However, the gap between dropout's training and inference phases, introduced due to tractability considerations, has largely remained under-appreciated. In this work, we first formulate dropout as a tractable approximation of some latent variable model, leading to a clean view of parameter sharing and enabling further theoretical analysis. Then, we introduce (approximate) expectation-linear dropout neural networks, whose inference gap we are able to formally characterize. Algorithmically, we show that our proposed measure of the inference gap can be used to regularize the standard dropout training objective, resulting in an explicit control of the gap. Our method is as simple and efficient as standard dropout. We further prove the upper bounds on the loss in accuracy due to expectation-linearization, describe classes of input distributions that expectation-linearize easily. Experiments on three image classification benchmark datasets demonstrate that reducing the inference gap can indeed improve the performance consistently.