# Theory of Deep Learning III: explaining the non-overfitting puzzle

arXiv: https://arxiv.org/abs/1801.00173 · HTML: https://mlsanity.com/paper/1801.00173/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2017.

## Built on by (most cited first, 30 of 32)

- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [PD-Quant: Post-Training Quantization based on Prediction Difference Metric](https://mlsanity.com/paper/2212.07048/) (2022)
- [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021)
- [Analytic Insights into Structure and Rank of Neural Network Hessian Maps](https://mlsanity.com/paper/2106.16225/) (2021)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [A deep learning theory for neural networks grounded in physics](https://mlsanity.com/paper/2103.09985/) (2021)
- [An SDE for Modeling SAM: Theory and Insights](https://mlsanity.com/paper/2301.08203/) (2023)
- [The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion](https://mlsanity.com/paper/2107.09133/) (2021)
- [Understanding Dynamics of Nonlinear Representation Learning and Its Application](https://mlsanity.com/paper/2106.14836/) (2021)
- [Provable Super-Convergence with a Large Cyclical Learning Rate](https://mlsanity.com/paper/2102.10734/) (2021)
- [A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks](https://mlsanity.com/paper/2305.09779/) (2023)
- [A Recipe for Global Convergence Guarantee in Deep Neural Networks](https://mlsanity.com/paper/2104.05785/) (2021)
- [Data Augmentation via Subgroup Mixup for Improving Fairness](https://mlsanity.com/paper/2309.07110/) (2023)
- [Model-agnostic Measure of Generalization Difficulty](https://mlsanity.com/paper/2305.01034/) (2023)
- [OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detection](https://mlsanity.com/paper/2504.17160/) (2025)
- [Gaussian Process Surrogate Models for Neural Networks](https://mlsanity.com/paper/2208.06028/) (2022)
- [Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs](https://mlsanity.com/paper/2502.17009/) (2025)
- [Learning distinct features helps, provably](https://mlsanity.com/paper/2106.06012/) (2021)
- [WLD-Reg: A Data-dependent Within-layer Diversity Regularizer](https://mlsanity.com/paper/2301.01352/) (2023)
- [Towards Unraveling and Improving Generalization in World Models](https://mlsanity.com/paper/2501.00195/) (2024)
- [A manifold learning approach for gesture recognition from micro-Doppler radar measurements](https://mlsanity.com/paper/2110.01670/) (2021)
- [Spectral Gating Networks](https://mlsanity.com/paper/2602.07679/) (2026)
- [Modeling Nonlinear Feature Interactions with Product-Unit Residual Networks](https://mlsanity.com/paper/2606.06861/) (2026)
- [To Boost or not to Boost: On the Limits of Boosted Neural Networks](https://mlsanity.com/paper/2107.13600/) (2021)
- [Similarity of Pre-trained and Fine-tuned Representations](https://mlsanity.com/paper/2207.09225/) (2022)
- [Deep Neural Networks as the Semi-classical Limit of Topological Quantum Neural Networks: The problem of generalisation](https://mlsanity.com/paper/2210.13741/) (2022)
- [Class-wise Activation Unravelling the Engima of Deep Double Descent](https://mlsanity.com/paper/2405.07679/) (2024)
- [Functional Risk Minimization](https://mlsanity.com/paper/2412.21149/) (2024)
- [The Urysohn Ladder: Recursive Metric Contraction for Scalable Continual Learning](https://mlsanity.com/paper/2512.18471/) (2025)

## Abstract (arXiv)

A main puzzle of deep networks revolves around the absence of overfitting despite large overparametrization and despite the large capacity demonstrated by zero training error on randomly labeled data. In this note, we show that the dynamics associated to gradient descent minimization of nonlinear networks is topologically equivalent, near the asymptotically stable minima of the empirical error, to linear gradient system in a quadratic potential with a degenerate (for square loss) or almost degenerate (for logistic or crossentropy loss) Hessian. The proposition depends on the qualitative theory of dynamical systems and is supported by numerical results. Our main propositions extend to deep nonlinear networks two properties of gradient descent for linear networks, that have been recently established (1) to be key to their generalization properties: 1. Gradient descent enforces a form of implicit regularization controlled by the number of iterations, and asymptotically converges to the minimum norm solution for appropriate initial conditions of gradient descent. This implies that there is usually an optimum early stopping that avoids overfitting of the loss. This property, valid for the square loss and many other loss functions, is relevant especially for regression. 2. For classification, the asymptotic convergence to the minimum norm solution implies convergence to the maximum margin solution which guarantees good classification error for "low noise" datasets. This property holds for loss functions such as the logistic and cross-entropy loss independently of the initial conditions. The robustness to overparametrization has suggestive implications for the robustness of the architecture of deep convolutional networks with respect to the curse of dimensionality.