# Generalization in Deep Learning

arXiv: https://arxiv.org/abs/1710.05468 · HTML: https://mlsanity.com/paper/1710.05468/

## Key facts

- Cited by 146 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2017.

## Built on by (most cited first, 30 of 146)

- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021)
- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?](https://mlsanity.com/paper/2109.09444/) (2021)
- [Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology](https://mlsanity.com/paper/2211.08939/) (2022)
- [TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo](https://mlsanity.com/paper/2111.07418/) (2021)
- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [Differentially Private Decoding in Large Language Models](https://mlsanity.com/paper/2205.13621/) (2022)
- [Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation](https://mlsanity.com/paper/2107.02629/) (2021)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization](https://mlsanity.com/paper/2303.00633/) (2023)
- [Robustness Implies Generalization via Data-Dependent Generalization Bounds](https://mlsanity.com/paper/2206.13497/) (2022)
- [Non-Vacuous Generalisation Bounds for Shallow Neural Networks](https://mlsanity.com/paper/2202.01627/) (2022)
- [Random matrix analysis of deep neural network weight matrices](https://mlsanity.com/paper/2203.14661/) (2022)
- [The dilemma of quantum neural networks](https://mlsanity.com/paper/2106.04975/) (2021)
- [Neuron Coverage-Guided Domain Generalization](https://mlsanity.com/paper/2103.00229/) (2021)
- [On Margins and Derandomisation in PAC-Bayes](https://mlsanity.com/paper/2107.03955/) (2021)
- [Variational Disentanglement for Domain Generalization](https://mlsanity.com/paper/2109.05826/) (2021)
- [Evaluating the Generalization Ability of Super-Resolution Networks](https://mlsanity.com/paper/2205.07019/) (2022)
- [Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration](https://mlsanity.com/paper/2409.19872/) (2024)
- [Limits for Learning with Language Models](https://mlsanity.com/paper/2306.12213/) (2023)
- [Can AI Be as Creative as Humans?](https://mlsanity.com/paper/2401.01623/) (2024)
- [A Manifold Perspective on the Statistical Generalization of Graph Neural Networks](https://mlsanity.com/paper/2406.05225/) (2024)
- [The no-free-lunch theorems of supervised learning](https://mlsanity.com/paper/2202.04513/) (2022)
- [On Catastrophic Inheritance of Large Foundation Models](https://mlsanity.com/paper/2402.01909/) (2024)
- [The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation](https://mlsanity.com/paper/2103.11488/) (2021)
- [Prediction Error-based Classification for Class-Incremental Learning](https://mlsanity.com/paper/2305.18806/) (2023)
- [Robustness to Augmentations as a Generalization metric](https://mlsanity.com/paper/2101.06459/) (2021)

## Abstract (arXiv)

This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature. We also discuss approaches to provide non-vacuous generalization guarantees for deep learning. Based on theoretical observations, we propose new open problems and discuss the limitations of our results.