# On the Regularization Properties of Structured Dropout

arXiv: https://arxiv.org/abs/1910.14186 · HTML: https://mlsanity.com/paper/1910.14186/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2019.

## Built on by (most cited first, 5 of 5)

- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [An Overview of Low-Rank Structures in the Training and Adaptation of Large Models](https://mlsanity.com/paper/2503.19859/) (2025)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction](https://mlsanity.com/paper/2206.00913/) (2022)
- [Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learning](https://mlsanity.com/paper/2510.01824/) (2025)

## Abstract (arXiv)

Dropout and its extensions (eg. DropBlock and DropConnect) are popular heuristics for training neural networks, which have been shown to improve generalization performance in practice. However, a theoretical understanding of their optimization and regularization properties remains elusive. Recent work shows that in the case of single hidden-layer linear networks, Dropout is a stochastic gradient descent method for minimizing a regularized loss, and that the regularizer induces solutions that are low-rank and balanced. In this work we show that for single hidden-layer linear networks, DropBlock induces spectral k-support norm regularization, and promotes solutions that are low-rank and have factors with equal norm. We also show that the global minimizer for DropBlock can be computed in closed form, and that DropConnect is equivalent to Dropout. We then show that some of these results can be extended to a general class of Dropout-strategies, and, with some assumptions, to deep non-linear networks when Dropout is applied to the last layer. We verify our theoretical claims and assumptions experimentally with commonly used network architectures.