# Generalization and Estimation Error Bounds for Model-based Neural Networks

arXiv: https://arxiv.org/abs/2304.09802 · HTML: https://mlsanity.com/paper/2304.09802/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 18 of 18)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Sensitivity and Generalization in Neural Networks: an Empirical Study](https://mlsanity.com/paper/1802.08760/) (2018)
- [Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks](https://mlsanity.com/paper/1805.12076/) (2018)
- [Robust Large Margin Deep Neural Networks](https://mlsanity.com/paper/1605.08254/) (2016)
- [Fisher-Rao Metric, Geometry, and Complexity of Neural Networks](https://mlsanity.com/paper/1711.01530/) (2017)
- [Generalization Error in Deep Learning](https://mlsanity.com/paper/1808.01174/) (2018)
- [Local Rademacher Complexity-based Learning Guarantees for Multi-Task Learning](https://mlsanity.com/paper/1602.05916/) (2016)
- [Compressive Sensing and Neural Networks from a Statistical Learning Perspective](https://mlsanity.com/paper/2010.15658/) (2020)
- [Dropout Rademacher Complexity of Deep Neural Networks](https://mlsanity.com/paper/1402.3811/) (2014)
- [Generalization Error Bounds for Iterative Recovery Algorithms Unfolded as Neural Networks](https://mlsanity.com/paper/2112.04364/) (2021)
- [Local Rademacher Complexity for Multi-label Learning](https://mlsanity.com/paper/1410.6990/) (2014)
- [An Empirical Study on Regularization of Deep Neural Networks by Local Rademacher Complexity](https://mlsanity.com/paper/1902.00873/) (2019)

## Built on by (most cited first, 6 of 6)

- [Deep Unfolding: Recent Developments, Theory, and Design Guidelines](https://mlsanity.com/paper/2512.03768/) (2025)
- [Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding](https://mlsanity.com/paper/2309.06195/) (2023)
- [Unrolled denoising networks provably learn optimal Bayesian inference](https://mlsanity.com/paper/2409.12947/) (2024)
- [On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective](https://mlsanity.com/paper/2602.20921/) (2026)
- [Theory Foundation of Physics-Enhanced Residual Learning](https://mlsanity.com/paper/2509.00348/) (2025)
- [Learning False Discovery Rate Control via Model-Based Neural Networks](https://mlsanity.com/paper/2602.05798/) (2026)

## Abstract (arXiv)

Model-based neural networks provide unparalleled performance for various tasks, such as sparse coding and compressed sensing problems. Due to the strong connection with the sensing model, these networks are interpretable and inherit prior structure of the problem. In practice, model-based neural networks exhibit higher generalization capability compared to ReLU neural networks. However, this phenomenon was not addressed theoretically. Here, we leverage complexity measures including the global and local Rademacher complexities, in order to provide upper bounds on the generalization and estimation errors of model-based networks. We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks, and derive practical design rules that allow to construct model-based networks with guaranteed high generalization. We demonstrate through a series of experiments that our theoretical insights shed light on a few behaviours experienced in practice, including the fact that ISTA and ADMM networks exhibit higher generalization abilities (especially for small number of training samples), compared to ReLU networks.