# A Priori Estimates of the Population Risk for Residual Networks

arXiv: https://arxiv.org/abs/1903.02154 · HTML: https://mlsanity.com/paper/1903.02154/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2019.

## Built on by (most cited first, 14 of 14)

- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [Towards Theoretical Understandings of Self-Consuming Generative Models](https://mlsanity.com/paper/2402.11778/) (2024)
- [Reproducing Activation Function for Deep Learning](https://mlsanity.com/paper/2101.04844/) (2021)
- [Generalization Error Bounds for Deep Neural Networks Trained by SGD](https://mlsanity.com/paper/2206.03299/) (2022)
- [Deep Network Approximation in Terms of Intrinsic Parameters](https://mlsanity.com/paper/2111.07964/) (2021)
- [Generalization Error of GAN from the Discriminator's Perspective](https://mlsanity.com/paper/2107.03633/) (2021)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality](https://mlsanity.com/paper/2209.01173/) (2022)
- [Why the pseudo label based semi-supervised learning algorithm is effective?](https://mlsanity.com/paper/2211.10039/) (2022)
- [Approximation properties of Residual Neural Networks for Kolmogorov PDEs](https://mlsanity.com/paper/2111.00215/) (2021)
- [Nonlinear Weighted Directed Acyclic Graph and A Priori Estimates for Neural Networks](https://mlsanity.com/paper/2103.16355/) (2021)
- [A priori Estimates for Deep Residual Network in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2402.16899/) (2024)
- [A First-Principles Theory of Slow Thinking and Active Perception](https://mlsanity.com/paper/2607.08196/) (2026)

## Abstract (arXiv)

Optimal a priori estimates are derived for the population risk, also known as the generalization error, of a regularized residual network model. An important part of the regularized model is the usage of a new path norm, called the weighted path norm, as the regularization term. The weighted path norm treats the skip connections and the nonlinearities differently so that paths with more nonlinearities are regularized by larger weights. The error estimates are a priori in the sense that the estimates depend only on the target function, not on the parameters obtained in the training process. The estimates are optimal, in a high dimensional setting, in the sense that both the bound for the approximation and estimation errors are comparable to the Monte Carlo error rates. A crucial step in the proof is to establish an optimal bound for the Rademacher complexity of the residual networks. Comparisons are made with existing norm-based generalization error bounds.