# An empirical analysis of dropout in piecewise linear networks

arXiv: https://arxiv.org/abs/1312.6197 · HTML: https://mlsanity.com/paper/1312.6197/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2013.

## Built on by (most cited first, 19 of 19)

- [LoRA Dropout as a Sparsity Regularizer for Overfitting Control](https://mlsanity.com/paper/2404.09610/) (2024)
- [DropoutGS: Dropping Out Gaussians for Better Sparse-view Rendering](https://mlsanity.com/paper/2504.09491/) (2025)
- [Revisiting Active Learning in the Era of Vision Foundation Models](https://mlsanity.com/paper/2401.14555/) (2024)
- [Few-shot Image Generation via Masked Discrimination](https://mlsanity.com/paper/2210.15194/) (2022)
- [Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2205.14557/) (2022)
- [Quaternion Factorization Machines: A Lightweight Solution to Intricate Feature Interaction Modelling](https://mlsanity.com/paper/2104.01716/) (2021)
- [Adaptive Stochastic Weight Averaging](https://mlsanity.com/paper/2406.19092/) (2024)
- [Burn After Reading: Online Adaptation for Cross-domain Streaming Data](https://mlsanity.com/paper/2112.04345/) (2021)
- [Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling](https://mlsanity.com/paper/2303.15973/) (2023)
- [Information Geometry of Dropout Training](https://mlsanity.com/paper/2206.10936/) (2022)
- [Enhancing Deep Hedging of Options with Implied Volatility Surface Feedback Information](https://mlsanity.com/paper/2407.21138/) (2024)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [Dropout Inference with Non-Uniform Weight Scaling](https://mlsanity.com/paper/2204.13047/) (2022)
- [Predicting Stock Price Movement as an Image Classification Problem](https://mlsanity.com/paper/2303.01111/) (2023)
- [Regularization Through Simultaneous Learning: A Case Study on Plant Classification](https://mlsanity.com/paper/2305.13447/) (2023)
- [Beyond Trend Following: Deep Learning for Market Trend Prediction](https://mlsanity.com/paper/2407.13685/) (2024)
- [Countering Overfitting with Counterfactual Examples](https://mlsanity.com/paper/2502.09193/) (2025)
- [Generalizability vs. Counterfactual Explainability Trade-Off](https://mlsanity.com/paper/2505.23225/) (2025)
- [Parameter Averaging in Link Prediction](https://mlsanity.com/paper/2510.25361/) (2025)

## Abstract (arXiv)

The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient.