# Approximation and Estimation for High-Dimensional Deep Learning Networks

arXiv: https://arxiv.org/abs/1809.03090 · HTML: https://mlsanity.com/paper/1809.03090/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2018.

## Built on by (most cited first, 22 of 22)

- [Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality](https://mlsanity.com/paper/2106.08900/) (2021)
- [Deep Network Approximation: Beyond ReLU to Diverse Activation Functions](https://mlsanity.com/paper/2307.06555/) (2023)
- [Formalizing Generalization and Robustness of Neural Networks to Weight Perturbations](https://mlsanity.com/paper/2103.02200/) (2021)
- [Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future](https://mlsanity.com/paper/2108.04543/) (2021)
- [Neural Network Architecture Beyond Width and Depth](https://mlsanity.com/paper/2205.09459/) (2022)
- [Theoretical analysis of deep neural networks for temporally dependent observations](https://mlsanity.com/paper/2210.11530/) (2022)
- [Provable and Efficient Continual Representation Learning](https://mlsanity.com/paper/2203.02026/) (2022)
- [Deep Network Approximation in Terms of Intrinsic Parameters](https://mlsanity.com/paper/2111.07964/) (2021)
- [Generalization Bound and New Algorithm for Clean-Label Backdoor Attack](https://mlsanity.com/paper/2406.00588/) (2024)
- [Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning](https://mlsanity.com/paper/2102.12328/) (2021)
- [On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network](https://mlsanity.com/paper/2301.12353/) (2023)
- [Feedback-driven recurrent quantum neural network universality](https://mlsanity.com/paper/2506.16332/) (2025)
- [Analytic function approximation by path norm regularized deep networks](https://mlsanity.com/paper/2104.02095/) (2021)
- [Nonlinear controllability and function representation by neural stochastic differential equations](https://mlsanity.com/paper/2212.00896/) (2022)
- [Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning](https://mlsanity.com/paper/2107.01081/) (2021)
- [Distance Metric-Based Learning with Interpolated Latent Features for Location Classification in Endoscopy Image and Video](https://mlsanity.com/paper/2103.08504/) (2021)
- [Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss](https://mlsanity.com/paper/2309.06774/) (2023)
- [A statistical perspective on algorithm unrolling models for inverse problems](https://mlsanity.com/paper/2311.06395/) (2023)
- [A priori Estimates for Deep Residual Network in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2402.16899/) (2024)
- [Generalizability of Neural Networks Minimizing Empirical Risk Based on Expressive Ability](https://mlsanity.com/paper/2503.04111/) (2025)
- [Random Neural Network Expressivity for Non-Linear Partial Differential Equations](https://mlsanity.com/paper/2605.25057/) (2026)
- [Quantifying and Optimizing Simplicity via Polynomial Representations](https://mlsanity.com/paper/2605.29823/) (2026)

## Abstract (arXiv)

It has been experimentally observed in recent years that multi-layer artificial neural networks have a surprising ability to generalize, even when trained with far more parameters than observations. Is there a theoretical basis for this? The best available bounds on their metric entropy and associated complexity measures are essentially linear in the number of parameters, which is inadequate to explain this phenomenon. Here we examine the statistical risk (mean squared predictive error) of multi-layer networks with ℓ1-type controls on their parameters and with ramp activation functions (also called lower-rectified linear units). In this setting, the risk is shown to be upper bounded by [(L3 log d)/n]1/2, where d is the input dimension to each layer, L is the number of layers, and n is the sample size. In this way, the input dimension can be much larger than the sample size and the estimator can still be accurate, provided the target function has such ℓ1 controls and that the sample size is at least moderately large compared to L3log d. The heart of the analysis is the development of a sampling strategy that demonstrates the accuracy of a sparse covering of deep ramp networks. Lower bounds show that the identified risk is close to being optimal.