# Dropout Neural Network Training Viewed from a Percolation Perspective

arXiv: https://arxiv.org/abs/2512.13853 · HTML: https://mlsanity.com/paper/2512.13853/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Dropout Training as Adaptive Regularization](https://mlsanity.com/paper/1307.1493/) (2013)
- [Deep Neural Network Approximation Theory](https://mlsanity.com/paper/1901.02220/) (2019)
- [Convergence rates for the stochastic gradient descent method for non-convex objective functions](https://mlsanity.com/paper/1904.01517/) (2019)
- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [On the Implicit Bias of Dropout](https://mlsanity.com/paper/1806.09777/) (2018)
- [On Convergence and Generalization of Dropout Training](https://mlsanity.com/paper/2010.12711/) (2020)
- [Masked Training of Neural Networks with Partial Gradients](https://mlsanity.com/paper/2106.08895/) (2021)
- [Regularization of Deep Neural Networks with Spectral Dropout](https://mlsanity.com/paper/1711.08591/) (2017)
- [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024)
- [Almost Sure Convergence of Dropout Algorithms for Neural Networks](https://mlsanity.com/paper/2002.02247/) (2020)
- [Asymptotic convergence rate of Dropout on shallow linear neural networks](https://mlsanity.com/paper/2012.01978/) (2020)
- [Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime](https://mlsanity.com/paper/2510.07554/) (2025)

## Built on by (most cited first, 2 of 2)

- [Critical Percolation as a Synthetic Data Model for Interpretability](https://mlsanity.com/paper/2606.20347/) (2026)
- [Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry](https://mlsanity.com/paper/2609.02373/) (2026)

## Abstract (arXiv)

In this work, we investigate the existence and effect of percolation in training deep Neural Networks (NNs) with dropout. Dropout methods are regularisation techniques for training NNs, first introduced by G. Hinton et al. (2012). These methods temporarily remove connections in the NN, randomly at each stage of training, and update the remaining subnetwork with Stochastic Gradient Descent (SGD). The process of removing connections from a network at random is similar to percolation, a paradigm model of statistical physics. If dropout were to remove enough connections such that there is no path between the input and output of the NN, then the NN could not make predictions informed by the data. We study new percolation models that mimic dropout in NNs and characterise the relationship between network topology and this path problem. The theory shows the existence of a percolative effect in dropout. We also show that this percolative effect can cause a breakdown when training NNs without biases with dropout; and we argue heuristically that this breakdown extends to NNs with biases.