# On Dropout and Nuclear Norm Regularization

arXiv: https://arxiv.org/abs/1905.11887 · HTML: https://mlsanity.com/paper/1905.11887/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 12 of 12)

- [Exact Solutions of a Deep Linear Network](https://mlsanity.com/paper/2202.04777/) (2022)
- [An Overview of Low-Rank Structures in the Training and Adaptation of Large Models](https://mlsanity.com/paper/2503.19859/) (2025)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Regularization in ResNet with Stochastic Depth](https://mlsanity.com/paper/2106.03091/) (2021)
- [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024)
- [Weight Expansion: A New Perspective on Dropout and Generalization](https://mlsanity.com/paper/2201.09209/) (2022)
- [Exact Phase Transitions in Deep Learning](https://mlsanity.com/paper/2205.12510/) (2022)
- [Stochastic Neural Networks with Infinite Width are Deterministic](https://mlsanity.com/paper/2201.12724/) (2022)
- [Flat Minima in Linear Estimation and an Extended Gauss Markov Theorem](https://mlsanity.com/paper/2311.11093/) (2023)
- [Overlap-Adaptive Regularization for Conditional Average Treatment Effect Estimation](https://mlsanity.com/paper/2509.24962/) (2025)
- [Estimating Implicit Regularization in Deep Learning](https://mlsanity.com/paper/2605.05436/) (2026)
- [Sample-Weighted End-to-End Trace-Norm Geometry for Multitask Learning](https://mlsanity.com/paper/2609.29520/) (2026)

## Abstract (arXiv)

We give a formal and complete characterization of the explicit regularizer induced by dropout in deep linear networks with squared loss. We show that (a) the explicit regularizer is composed of an ℓ2-path regularizer and other terms that are also re-scaling invariant, (b) the convex envelope of the induced regularizer is the squared nuclear norm of the network map, and (c) for a sufficiently large dropout rate, we characterize the global optima of the dropout objective. We validate our theoretical findings with empirical results.