# Convergence of Deep Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2109.13542 · HTML: https://mlsanity.com/paper/2109.13542/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Universal approximations of invariant maps by neural networks](https://mlsanity.com/paper/1804.10306/) (2018)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators](https://mlsanity.com/paper/1903.01014/) (2019)
- [The gap between theory and practice in function approximation with deep neural networks](https://mlsanity.com/paper/2001.07523/) (2020)
- [Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theorem](https://mlsanity.com/paper/1906.11945/) (2019)
- [Nonlinear Approximation and (Deep) ReLU Networks](https://mlsanity.com/paper/1905.02199/) (2019)
- [Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions](https://mlsanity.com/paper/1807.00297/) (2018)
- [Equivalence of approximation by convolutional neural networks and fully-connected networks](https://mlsanity.com/paper/1809.00973/) (2018)
- [Convergence of Deep ReLU Networks](https://mlsanity.com/paper/2107.12530/) (2021)
- [A Mathematical Introduction to Generative Adversarial Nets (GAN)](https://mlsanity.com/paper/2009.00169/) (2020)

## Built on by (most cited first, 13 of 13)

- [Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2307.01998/) (2023)
- [Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning](https://mlsanity.com/paper/2410.16105/) (2024)
- [Multi-Grade Deep Learning](https://mlsanity.com/paper/2302.00150/) (2023)
- [Sparse Deep Neural Network for Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2207.13266/) (2022)
- [Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths](https://mlsanity.com/paper/2306.01692/) (2023)
- [Multi-Grade Deep Learning for Partial Differential Equations with Applications to the Burgers Equation](https://mlsanity.com/paper/2309.07401/) (2023)
- [Deep Neural Network Solutions for Oscillatory Fredholm Integral Equations](https://mlsanity.com/paper/2401.07003/) (2024)
- [Successive Affine Learning for Deep Neural Networks](https://mlsanity.com/paper/2305.07996/) (2023)
- [Hypothesis Spaces for Deep Learning](https://mlsanity.com/paper/2403.03353/) (2024)
- [Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study](https://mlsanity.com/paper/2410.11279/) (2024)
- [Convergence Analysis of Deep Residual Networks](https://mlsanity.com/paper/2205.06571/) (2022)
- [Convergence of Deep Neural Networks with General Activation Functions and Pooling](https://mlsanity.com/paper/2205.06570/) (2022)
- [Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence](https://mlsanity.com/paper/2609.25874/) (2026)

## Abstract (arXiv)

Convergence of deep neural networks as the depth of the networks tends to infinity is fundamental in building the mathematical foundation for deep learning. In a previous study, we investigated this question for deep ReLU networks with a fixed width. This does not cover the important convolutional neural networks where the widths are increasing from layer to layer. For this reason, we first study convergence of general ReLU networks with increasing widths and then apply the results obtained to deep convolutional neural networks. It turns out the convergence reduces to convergence of infinite products of matrices with increasing sizes, which has not been considered in the literature. We establish sufficient conditions for convergence of such infinite products of matrices. Based on the conditions, we present sufficient conditions for piecewise convergence of general deep ReLU networks with increasing widths, and as well as pointwise convergence of deep ReLU convolutional neural networks.