# Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness

arXiv: https://arxiv.org/abs/1905.11427 · HTML: https://mlsanity.com/paper/1905.11427/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 22 of 22)

- [Physics-informed neural networks with hard constraints for inverse design](https://mlsanity.com/paper/2102.04626/) (2021)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Reliable extrapolation of deep neural operators informed by physics or sparse observations](https://mlsanity.com/paper/2212.06347/) (2022)
- [Embedding Principle of Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2105.14573/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domains](https://mlsanity.com/paper/2104.10873/) (2021)
- [Towards Understanding the Condensation of Neural Networks at Initial Training](https://mlsanity.com/paper/2105.11686/) (2021)
- [Embedding Principle: a hierarchical structure of loss landscape of deep neural networks](https://mlsanity.com/paper/2111.15527/) (2021)
- [A Contract Theory based Incentive Mechanism for Federated Learning](https://mlsanity.com/paper/2108.05568/) (2021)
- [Certified machine learning: Rigorous a posteriori error bounds for PDE defined PINNs](https://mlsanity.com/paper/2210.03426/) (2022)
- [Approximation capabilities of measure-preserving neural networks](https://mlsanity.com/paper/2106.10911/) (2021)
- [Protecting Split Learning by Potential Energy Loss](https://mlsanity.com/paper/2210.09617/) (2022)
- [PDE+: Enhancing Generalization via PDE with Adaptive Distributional Diffusion](https://mlsanity.com/paper/2305.15835/) (2023)
- [Learning Functional Transduction](https://mlsanity.com/paper/2302.00328/) (2023)
- [Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchy](https://mlsanity.com/paper/2502.12125/) (2025)
- [Runtime Safety Monitoring of Deep Neural Networks for Perception: A Survey](https://mlsanity.com/paper/2511.05982/) (2025)
- [On Theory-training Neural Networks to Infer the Solution of Highly Coupled Differential Equations](https://mlsanity.com/paper/2102.04890/) (2021)
- [Scarce Data Driven Deep Learning of Drones via Generalized Data Distribution Space](https://mlsanity.com/paper/2108.08244/) (2021)
- [Collaborative Learning for Annotation-Efficient Volumetric MR Image Segmentation](https://mlsanity.com/paper/2312.10978/) (2023)
- [Limitation of Characterizing Implicit Regularization by Data-independent Functions](https://mlsanity.com/paper/2201.12198/) (2022)
- [Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes](https://mlsanity.com/paper/2308.00858/) (2023)
- [Deep Learning Classification of EEG Responses to Multi-Dimensional Transcranial Electrical Stimulation](https://mlsanity.com/paper/2512.20319/) (2025)

## Abstract (arXiv)

The accuracy of deep learning, i.e., deep neural networks, can be characterized by dividing the total error into three main types: approximation error, optimization error, and generalization error. Whereas there are some satisfactory answers to the problems of approximation and optimization, much less is known about the theory of generalization. Most existing theoretical works for generalization fail to explain the performance of neural networks in practice. To derive a meaningful bound, we study the generalization error of neural networks for classification problems in terms of data distribution and neural network smoothness. We introduce the cover complexity (CC) to measure the difficulty of learning a data set and the inverse of the modulus of continuity to quantify neural network smoothness. A quantitative bound for expected accuracy/error is derived by considering both the CC and neural network smoothness. Although most of the analysis is general and not specific to neural networks, we validate our theoretical assumptions and results numerically for neural networks by several data sets of images. The numerical results confirm that the expected error of trained networks scaled with the square root of the number of classes has a linear relationship with respect to the CC. We also observe a clear consistency between test loss and neural network smoothness during the training process. In addition, we demonstrate empirically that the neural network smoothness decreases when the network size increases whereas the smoothness is insensitive to training dataset size.