mlsanityData to 7 Oct 2026

Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration

arXiv 2110.00653 ↗First posted 1 Oct 2021stat.ML cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Deep learning has powered recent successes of artificial intelligence (AI). However, the deep neural network, as the basic model of deep learning, has suffered from issues such as local traps and miscalibration. In this paper, we provide a new framework for sparse deep learning, which has the above issues addressed in a coherent way. In particular, we lay down a theoretical foundation for sparse deep learning and propose prior annealing algorithms for learning sparse neural networks. The former has successfully tamed the sparse deep neural network into the framework of statistical modeling, enabling prediction uncertainty correctly quantified. The latter can be asymptotically guaranteed to converge to the global optimum, enabling the validity of the down-stream statistical inference. Numerical result indicates the superiority of the proposed method compared to the existing ones.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

25 of 36 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  9. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  10. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  11. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  12. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
13 more
  1. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  2. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  3. 2020Dynamic Model Pruning with Feedbackcited by 125
  4. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  5. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 90
  6. 2017The loss surface of deep and wide neural networkscited by 74
  7. 2019Stochastic gradient Markov chain Monte Carlocited by 56
  8. 2018Posterior Concentration for Sparse Deep Learningcited by 50
  9. 2017Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classificationcited by 39
  10. 2016On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integratorscited by 35
  11. 2021Consistent Sparse Deep Learning: Theory and Computationcited by 27
  12. 2020Uncertainty Quantification for Sparse Deep Learningcited by 13
  13. 2013Variable selection for BART: An application to gene regulationcited by 10

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 3; 2025: 1.

7 papers.

  1. 2022Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constantscited by 24
  2. 2024Investigating Calibration and Corruption Robustness of Post-hoc Pruned Perception CNNs: An Image Classification Benchmark Studycited by 5
  3. 2023Sparse Deep Learning for Time Series Data: Theory and Applicationscited by 3
  4. 2024Causal-StoNet: Causal Inference for High-Dimensional Complex Datacited by 3
  5. 2022Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networkscited by 1
  6. 2024Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Priorcited by 1
  7. 2025Uncertainty Quantification for Large-Scale Deep Networks via Post-StoNet Modelingcited by 0