mlsanityData to 14 Oct 2026

Statistical guarantees for sparse deep learning

arXiv 2212.05427 ↗First posted 11 Dec 2022cs.LG cs.AI math.STcited by 9

Not on the map. Why?

Abstract (arXiv)

Neural networks are becoming increasingly popular in applications, but our mathematical understanding of their potential and limitations is still limited. In this paper, we further this understanding by developing statistical guarantees for sparse deep learning. In contrast to previous work, we consider different types of sparsity, such as few active connections, few active nodes, and other norm-based types of sparsity. Moreover, our theories cover important aspects that previous theories have neglected, such as multiple outputs, regularization, and l2-loss. The guarantees have a mild dependence on network widths and depths, which means that they support the application of sparse but wide and deep networks from a statistical perspective. Some of the concepts and tools that we use in our derivations are uncommon in deep learning and, hence, might be of additional interest.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 26 of its 60 references that are in the map. First posted on arXiv 11 Dec 2022.

Builds on

26 of 60 references.

References · 26

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  7. 2016Error bounds for approximations with deep ReLU networkscited by 574
  8. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  9. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  10. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  11. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  12. 2016Benefits of depth in neural networkscited by 250
14 more
  1. 2021Accelerating Sparse Deep Neural Networkscited by 223
  2. 2015All you need is a good initcited by 124
  3. 2016Group Sparse Regularization for Deep Neural Networkscited by 89
  4. 2016Why Deep Neural Networks for Function Approximation?cited by 82
  5. 2016Learning the Number of Neurons in Deep Networkscited by 69
  6. 2017Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classificationcited by 39
  7. 2020Statistical Guarantees for Regularized Neural Networkscited by 25
  8. 2014On the Prediction Performance of the Lassocited by 12
  9. 2020Risk Bounds for Robust Deep Learningcited by 9
  10. 2020Layer Sparsity in Neural Networkscited by 6
  11. 2016Oracle Inequalities for High-dimensional Predictioncited by 4
  12. 2020Estimating the Lasso's Effective Noisecited by 2
  13. 2021Function approximation by deep neural networks with parameters 0,± frac12, ± 1, 2cited by 2
  14. 2017Maximum Regularized Likelihood Estimators: A General Prediction Theory and Applicationscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 2; 2025: 0; 2026: 2.

9 papers.

  1. 2023Single-Model Attribution of Generative Models Through Final-Layer Inversioncited by 10
  2. 2023Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGDcited by 5
  3. 2024How many samples are needed to train a deep neural network?cited by 5
  4. 2023Sparse-Input Neural Network using Group Concave Regularizationcited by 4
  5. 2023Online Detection of Changes in Moment-Based Projections: When to Retrain Deep Learners or Update Portfolios?cited by 2
  6. 2023Statistical learning by sparse deep neural networkscited by 1
  7. 2024Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regressioncited by 0
  8. 2026Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decodingcited by 0
  9. 2026Stein-Encoder: A White-Box Supervised Encoder via Stein Identities in Multi-Modal Studiescited by 0