# Approximation Properties of Deep ReLU CNNs

arXiv: https://arxiv.org/abs/2109.00190 · HTML: https://mlsanity.com/paper/2109.00190/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [Characterization of the Variation Spaces Corresponding to Shallow Neural Networks](https://mlsanity.com/paper/2106.15002/) (2021)
- [Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks](https://mlsanity.com/paper/1903.10047/) (2019)
- [Equivalence of approximation by convolutional neural networks and fully-connected networks](https://mlsanity.com/paper/1809.00973/) (2018)
- [Nonlinear Approximation via Compositions](https://mlsanity.com/paper/1902.10170/) (2019)
- [Universal Consistency of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2106.12498/) (2021)
- [Universal Approximation Theorem for Equivariant Maps by Group CNNs](https://mlsanity.com/paper/2012.13882/) (2020)
- [ReLU Deep Neural Networks from the Hierarchical Basis Perspective](https://mlsanity.com/paper/2105.04156/) (2021)
- [Constrained Linear Data-feature Mapping for Image Classification](https://mlsanity.com/paper/1911.10428/) (2019)

## Built on by (most cited first, 15 of 15)

- [MgNO: Efficient Parameterization of Linear Operators via Multigrid](https://mlsanity.com/paper/2310.19809/) (2023)
- [Theoretical Analysis of Inductive Biases in Deep Convolutional Networks](https://mlsanity.com/paper/2305.08404/) (2023)
- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activation](https://mlsanity.com/paper/2312.16483/) (2023)
- [An Interpretive Constrained Linear Model for ResNet and MgNet](https://mlsanity.com/paper/2112.07441/) (2021)
- [Approximation analysis of CNNs from a feature extraction view](https://mlsanity.com/paper/2210.09041/) (2022)
- [Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms](https://mlsanity.com/paper/2508.05141/) (2025)
- [An Enhanced V-cycle MgNet Model for Operator Learning in Numerical Partial Differential Equations](https://mlsanity.com/paper/2302.00938/) (2023)
- [Higher Order Approximation Rates for ReLU CNNs in Korobov Spaces](https://mlsanity.com/paper/2501.11275/) (2025)
- [FV-MgNet: Fully Connected V-cycle MgNet for Interpretable Time Series Forecasting](https://mlsanity.com/paper/2302.00962/) (2023)
- [Two-Dimensional Deep ReLU CNN Approximation for Korobov Functions: A Constructive Approach](https://mlsanity.com/paper/2503.07976/) (2025)
- [Optimal Convergence Rates of Deep Convolutional Neural Networks: Additive Ridge Functions](https://mlsanity.com/paper/2202.12119/) (2022)
- [Balanced Group Convolution: An Improved Group Convolution Based on Approximability Estimates](https://mlsanity.com/paper/2310.12461/) (2023)
- [Progressive Approximation in Deep Residual Networks: Theory and Validation](https://mlsanity.com/paper/2604.24154/) (2026)
- [Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations](https://mlsanity.com/paper/2605.22557/) (2026)

## Abstract (arXiv)

This paper focuses on establishing L2 approximation properties for deep ReLU convolutional neural networks (CNNs) in two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels with a large spatial size and multi-channels. Given the decomposition result, the property of the ReLU activation function, and a specific structure for channels, a universal approximation theorem of deep ReLU CNNs with classic structure is obtained by showing its connection with one-hidden-layer ReLU neural networks (NNs). Furthermore, approximation properties are obtained for one version of neural networks with ResNet, pre-act ResNet, and MgNet architecture based on connections between these networks.