# Information Geometry of Dropout Training

arXiv: https://arxiv.org/abs/2206.10936 · HTML: https://mlsanity.com/paper/2206.10936/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 22 Jun 2022.

## Builds on (most cited first, 19 of 19)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [A Kronecker-factored approximate Fisher matrix for convolution layers](https://mlsanity.com/paper/1602.01407/) (2016)
- [Survey of Dropout Methods for Deep Neural Networks](https://mlsanity.com/paper/1904.13310/) (2019)
- [Dropout improves Recurrent Neural Networks for Handwriting Recognition](https://mlsanity.com/paper/1312.4569/) (2013)
- [Riemannian metrics for neural networks I: feedforward networks](https://mlsanity.com/paper/1303.0818/) (2013)
- [An empirical analysis of dropout in piecewise linear networks](https://mlsanity.com/paper/1312.6197/) (2013)
- [On the Inductive Bias of Dropout](https://mlsanity.com/paper/1412.4736/) (2014)
- [Towards Dropout Training for Convolutional Neural Networks](https://mlsanity.com/paper/1512.00242/) (2015)
- [Fraternal Dropout](https://mlsanity.com/paper/1711.00066/) (2017)
- [Guided Dropout](https://mlsanity.com/paper/1812.03965/) (2018)
- [Information Geometry Connecting Wasserstein Distance and Kullback-Leibler Divergence via the Entropy-Relaxed Transportation Problem](https://mlsanity.com/paper/1709.10219/) (2017)
- [DropCluster: A structured dropout for convolutional networks](https://mlsanity.com/paper/2002.02997/) (2020)
- [α-Geodesical Skew Divergence](https://mlsanity.com/paper/2103.17060/) (2021)
- [Dropout with Tabu Strategy for Regularizing Deep Neural Networks](https://mlsanity.com/paper/1808.09907/) (2018)

## Built on by (most cited first, 2 of 2)

- [MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection](https://mlsanity.com/paper/2505.11416/) (2025)
- [Fashion-Specific Attributes Interpretation via Dual Gaussian Visual-Semantic Embedding](https://mlsanity.com/paper/2210.17417/) (2022)

## Abstract (arXiv)

Dropout is one of the most popular regularization techniques in neural network training. Because of its power and simplicity of idea, dropout has been analyzed extensively and many variants have been proposed. In this paper, several properties of dropout are discussed in a unified manner from the viewpoint of information geometry. We showed that dropout flattens the model manifold and that their regularization performance depends on the amount of the curvature. Then, we showed that dropout essentially corresponds to a regularization that depends on the Fisher information, and support this result from numerical experiments. Such a theoretical analysis of the technique from a different perspective is expected to greatly assist in the understanding of neural networks, which are still in their infancy.