# Understanding Generalization and Optimization Performance of Deep CNNs

arXiv: https://arxiv.org/abs/1805.10767 · HTML: https://mlsanity.com/paper/1805.10767/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2018.

## Built on by (most cited first, 9 of 9)

- [Analyzing Overfitting under Class Imbalance in Neural Networks for Image Segmentation](https://mlsanity.com/paper/2102.10365/) (2021)
- [Evaluating the Generalization Ability of Super-Resolution Networks](https://mlsanity.com/paper/2205.07019/) (2022)
- [Generalization Error Bounds for Deep Neural Networks Trained by SGD](https://mlsanity.com/paper/2206.03299/) (2022)
- [A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning](https://mlsanity.com/paper/2106.14749/) (2021)
- [Leaner Transformers: More Heads, Less Depth](https://mlsanity.com/paper/2505.20802/) (2025)
- [Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2105.00292/) (2021)
- [CNNs Avoid Curse of Dimensionality by Learning on Patches](https://mlsanity.com/paper/2205.10760/) (2022)
- [Slope and generalization properties of neural networks](https://mlsanity.com/paper/2107.01473/) (2021)
- [Multi-scale Fusion Fault Diagnosis Method Based on Two-Dimensionaliztion Sequence in Complex Scenarios](https://mlsanity.com/paper/2304.05198/) (2023)

## Abstract (arXiv)

This work aims to provide understandings on the remarkable success of deep convolutional neural networks (CNNs) by theoretically analyzing their generalization performance and establishing optimization guarantees for gradient descent based training algorithms. Specifically, for a CNN model consisting of l convolutional layers and one fully connected layer, we prove that its generalization error is bounded by O(√dtwidetildevarrho/n) where θ denotes freedom degree of the network parameters and widetildevarrho=O(log(prodi=1lrwii (kii-sii+1)/p)+log(rf)) encapsulates architecture parameters including the kernel size kii, stride sii, pooling size p and parameter magnitude rwii. To our best knowledge, this is the first generalization bound that only depends on O(log(prodi=1l+1rwii)), tighter than existing ones that all involve an exponential term like O(prodi=1l+1rwii). Besides, we prove that for an arbitrary gradient descent algorithm, the computed approximate stationary point by minimizing empirical risk is also an approximate stationary point to the population risk. This well explains why gradient descent training algorithms usually perform sufficiently well in practice. Furthermore, we prove the one-to-one correspondence and convergence guarantees for the non-degenerate stationary points between the empirical and population risks. It implies that the computed local minimum for the empirical risk is also close to a local minimum for the population risk, thus ensuring the good generalization performance of CNNs.