# Generalization Error in Deep Learning

arXiv: https://arxiv.org/abs/1808.01174 · HTML: https://mlsanity.com/paper/1808.01174/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2018.

## Built on by (most cited first, 30 of 47)

- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [On the Stability of Iterative Retraining of Generative Models on their own Data](https://mlsanity.com/paper/2310.00429/) (2023)
- [Self-Correcting Self-Consuming Loops for Generative Model Training](https://mlsanity.com/paper/2402.07087/) (2024)
- [Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equation](https://mlsanity.com/paper/2301.11509/) (2023)
- [Deep Learning and Symbolic Regression for Discovering Parametric Equations](https://mlsanity.com/paper/2207.00529/) (2022)
- [Generalization bounds for graph convolutional neural networks via Rademacher complexity](https://mlsanity.com/paper/2102.10234/) (2021)
- [On Rademacher Complexity-based Generalization Bounds for Deep Learning](https://mlsanity.com/paper/2208.04284/) (2022)
- [Bounding the Rademacher Complexity of Fourier neural operators](https://mlsanity.com/paper/2209.05150/) (2022)
- [Unsupervised Layer-wise Score Aggregation for Textual OOD Detection](https://mlsanity.com/paper/2302.09852/) (2023)
- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection](https://mlsanity.com/paper/2203.14380/) (2022)
- [Topology and geometry of data manifold in deep learning](https://mlsanity.com/paper/2204.08624/) (2022)
- [Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL](https://mlsanity.com/paper/2209.09845/) (2022)
- [Learning-based Design of Luenberger Observers for Autonomous Nonlinear Systems](https://mlsanity.com/paper/2210.01476/) (2022)
- [Approximating Numerical Fluxes Using Fourier Neural Operators for Hyperbolic Conservation Laws](https://mlsanity.com/paper/2401.01783/) (2024)
- [Rate-Distortion Optimized Post-Training Quantization for Learned Image Compression](https://mlsanity.com/paper/2211.02854/) (2022)
- [Information-Theoretic Generalization Bounds for Deep Neural Networks](https://mlsanity.com/paper/2404.03176/) (2024)
- [Generalization Error Bounds on Deep Learning with Markov Datasets](https://mlsanity.com/paper/2201.11059/) (2021)
- [Universal Novelty Detection Through Adaptive Contrastive Learning](https://mlsanity.com/paper/2408.10798/) (2024)
- [Generalization Error Bound for Quantum Machine Learning in NISQ Era -- A Survey](https://mlsanity.com/paper/2409.07626/) (2024)
- [Generalization and Estimation Error Bounds for Model-based Neural Networks](https://mlsanity.com/paper/2304.09802/) (2023)
- [Deep neural networks architectures from the perspective of manifold learning](https://mlsanity.com/paper/2306.03406/) (2023)
- [Combining Varied Learners for Binary Classification using Stacked Generalization](https://mlsanity.com/paper/2202.08910/) (2022)
- [Global Convergence Rate of Deep Equilibrium Models with General Activations](https://mlsanity.com/paper/2302.05797/) (2023)
- [Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding](https://mlsanity.com/paper/2309.06195/) (2023)
- [HierMUD: Hierarchical Multi-task Unsupervised Domain Adaptation between Bridges for Drive-by Damage Diagnosis](https://mlsanity.com/paper/2107.11435/) (2021)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [Lipschitzness Effect of a Loss Function on Generalization Performance of Deep Neural Networks Trained by Adam and AdamW Optimizers](https://mlsanity.com/paper/2303.16464/) (2023)
- [Understanding and Leveraging the Learning Phases of Neural Networks](https://mlsanity.com/paper/2312.06887/) (2023)
- [Neural Operators Learn the Local Physics of Magnetohydrodynamics](https://mlsanity.com/paper/2404.16015/) (2024)

## Abstract (arXiv)

Deep learning models have lately shown great performance in various fields such as computer vision, speech recognition, speech translation, and natural language processing. However, alongside their state-of-the-art performance, it is still generally unclear what is the source of their generalization ability. Thus, an important question is what makes deep neural networks able to generalize well from the training set to new data. In this article, we provide an overview of the existing theory and bounds for the characterization of the generalization error of deep neural networks, combining both classical and more recent theoretical and empirical results.