# Approximation bounds for norm constrained neural networks with applications to regression and GANs

arXiv: https://arxiv.org/abs/2201.09418 · HTML: https://mlsanity.com/paper/2201.09418/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 63 references that are in the map.
- First posted on arXiv 24 Jan 2022.

## Builds on (most cited first, 30 of 35)

- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Optimal approximation of continuous functions by very deep ReLU networks](https://mlsanity.com/paper/1802.03620/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Path-SGD: Path-Normalized Optimization in Deep Neural Networks](https://mlsanity.com/paper/1506.02617/) (2015)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [The Kolmogorov-Arnold representation theorem revisited](https://mlsanity.com/paper/2007.15884/) (2020)
- [The phase diagram of approximation rates for deep neural networks](https://mlsanity.com/paper/1906.09477/) (2019)
- [How Well Generative Adversarial Networks Learn Distributions](https://mlsanity.com/paper/1811.03179/) (2018)
- [On the regularization of Wasserstein GANs](https://mlsanity.com/paper/1709.08894/) (2017)
- [Limitations of the Lipschitz constant as a defense against adversarial examples](https://mlsanity.com/paper/1807.09705/) (2018)
- [On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths](https://mlsanity.com/paper/2101.09612/) (2021)
- [An error analysis of generative adversarial networks for learning distributions](https://mlsanity.com/paper/2105.13010/) (2021)
- [Sobolev GAN](https://mlsanity.com/paper/1711.04894/) (2017)
- [Nonparametric Density Estimation under Adversarial Losses](https://mlsanity.com/paper/1805.08836/) (2018)

## Built on by (most cited first, 22 of 22)

- [Nonparametric regression using over-parameterized shallow ReLU neural networks](https://mlsanity.com/paper/2306.08321/) (2023)
- [Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regression](https://mlsanity.com/paper/2304.01561/) (2023)
- [On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks](https://mlsanity.com/paper/2409.00901/) (2024)
- [On the rates of convergence for learning with convolutional neural networks](https://mlsanity.com/paper/2403.16459/) (2024)
- [Deep Transfer Learning: Model Framework and Error Analysis](https://mlsanity.com/paper/2410.09383/) (2024)
- [Wasserstein Distributionally Robust Nonparametric Regression](https://mlsanity.com/paper/2505.07967/) (2025)
- [Some Super-approximation Rates of ReLU Neural Networks for Korobov Functions](https://mlsanity.com/paper/2507.10345/) (2025)
- [Learning Sparse Compositional Functions with Norm-Constrained Neural Networks](https://mlsanity.com/paper/2605.25608/) (2026)
- [Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks](https://mlsanity.com/paper/2605.31152/) (2026)
- [Non-Asymptotic Bounds for Adversarial Excess Risk under Misspecified Models](https://mlsanity.com/paper/2309.00771/) (2023)
- [Operator Learning for Families of Finite-State Mean-Field Games](https://mlsanity.com/paper/2602.13169/) (2026)
- [Learning Distributions by Generative Adversarial Networks: Approximation and Generalization](https://mlsanity.com/paper/2205.12601/) (2022)
- [Neural Network Approximation for Pessimistic Offline Reinforcement Learning](https://mlsanity.com/paper/2312.11863/) (2023)
- [Theoretical Insights into CycleGAN: Analyzing Approximation and Estimation Errors in Unpaired Data Generation](https://mlsanity.com/paper/2407.11678/) (2024)
- [Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guarantees](https://mlsanity.com/paper/2408.08533/) (2024)
- [Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernels](https://mlsanity.com/paper/2511.05585/) (2025)
- [Distribution estimation and change-point estimation for time series via DNN-based GANs](https://mlsanity.com/paper/2211.14577/) (2022)
- [Consistency for Large Neural Networks: Regression and Classification](https://mlsanity.com/paper/2409.14123/) (2024)
- [Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width](https://mlsanity.com/paper/2607.10589/) (2026)
- [Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks](https://mlsanity.com/paper/2609.19937/) (2026)
- [Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence](https://mlsanity.com/paper/2609.25874/) (2026)
- [Joint Contrastive-Adversarial Representation Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2610.15613/) (2026)

## Abstract (arXiv)

This paper studies the approximation capacity of ReLU neural networks with norm constraint on the weights. We prove upper and lower bounds on the approximation error of these networks for smooth function classes. The lower bound is derived through the Rademacher complexity of neural networks, which may be of independent interest. We apply these approximation bounds to analyze the convergences of regression using norm constrained neural networks and distribution estimation by GANs. In particular, we obtain convergence rates for over-parameterized neural networks. It is also shown that GANs can achieve optimal rate of learning probability distributions, when the discriminator is a properly chosen norm constrained neural network.