# Statistical guarantees for sparse deep learning

arXiv: https://arxiv.org/abs/2212.05427 · HTML: https://mlsanity.com/paper/2212.05427/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 60 references that are in the map.
- First posted on arXiv 11 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Group Sparse Regularization for Deep Neural Networks](https://mlsanity.com/paper/1607.00485/) (2016)
- [Why Deep Neural Networks for Function Approximation?](https://mlsanity.com/paper/1610.04161/) (2016)
- [Learning the Number of Neurons in Deep Networks](https://mlsanity.com/paper/1611.06321/) (2016)
- [Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification](https://mlsanity.com/paper/1711.07592/) (2017)
- [Statistical Guarantees for Regularized Neural Networks](https://mlsanity.com/paper/2006.00294/) (2020)
- [On the Prediction Performance of the Lasso](https://mlsanity.com/paper/1402.1700/) (2014)
- [Risk Bounds for Robust Deep Learning](https://mlsanity.com/paper/2009.06202/) (2020)
- [Layer Sparsity in Neural Networks](https://mlsanity.com/paper/2006.15604/) (2020)
- [Oracle Inequalities for High-dimensional Prediction](https://mlsanity.com/paper/1608.00624/) (2016)
- [Estimating the Lasso's Effective Noise](https://mlsanity.com/paper/2004.11554/) (2020)
- [Function approximation by deep neural networks with parameters 0,± frac12, ± 1, 2](https://mlsanity.com/paper/2103.08659/) (2021)
- [Maximum Regularized Likelihood Estimators: A General Prediction Theory and Applications](https://mlsanity.com/paper/1710.02950/) (2017)

## Built on by (most cited first, 9 of 9)

- [Single-Model Attribution of Generative Models Through Final-Layer Inversion](https://mlsanity.com/paper/2306.06210/) (2023)
- [Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD](https://mlsanity.com/paper/2306.08125/) (2023)
- [How many samples are needed to train a deep neural network?](https://mlsanity.com/paper/2405.16696/) (2024)
- [Sparse-Input Neural Network using Group Concave Regularization](https://mlsanity.com/paper/2307.00344/) (2023)
- [Online Detection of Changes in Moment-Based Projections: When to Retrain Deep Learners or Update Portfolios?](https://mlsanity.com/paper/2302.07198/) (2023)
- [Statistical learning by sparse deep neural networks](https://mlsanity.com/paper/2311.08845/) (2023)
- [Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression](https://mlsanity.com/paper/2406.18137/) (2024)
- [Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decoding](https://mlsanity.com/paper/2603.21042/) (2026)
- [Stein-Encoder: A White-Box Supervised Encoder via Stein Identities in Multi-Modal Studies](https://mlsanity.com/paper/2605.25734/) (2026)

## Abstract (arXiv)

Neural networks are becoming increasingly popular in applications, but our mathematical understanding of their potential and limitations is still limited. In this paper, we further this understanding by developing statistical guarantees for sparse deep learning. In contrast to previous work, we consider different types of sparsity, such as few active connections, few active nodes, and other norm-based types of sparsity. Moreover, our theories cover important aspects that previous theories have neglected, such as multiple outputs, regularization, and l2-loss. The guarantees have a mild dependence on network widths and depths, which means that they support the application of sparse but wide and deep networks from a statistical perspective. Some of the concepts and tools that we use in our derivations are uncommon in deep learning and, hence, might be of additional interest.