# Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration

arXiv: https://arxiv.org/abs/2110.00653 · HTML: https://mlsanity.com/paper/2110.00653/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization](https://mlsanity.com/paper/1902.05967/) (2019)
- [A Complete Recipe for Stochastic Gradient MCMC](https://mlsanity.com/paper/1506.04696/) (2015)
- [Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks](https://mlsanity.com/paper/1811.08888/) (2018)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [An Improved Analysis of Training Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/1906.04688/) (2019)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [The loss surface of deep and wide neural networks](https://mlsanity.com/paper/1704.08045/) (2017)
- [Stochastic gradient Markov chain Monte Carlo](https://mlsanity.com/paper/1907.06986/) (2019)
- [Posterior Concentration for Sparse Deep Learning](https://mlsanity.com/paper/1803.09138/) (2018)
- [Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification](https://mlsanity.com/paper/1711.07592/) (2017)
- [On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators](https://mlsanity.com/paper/1610.06665/) (2016)
- [Consistent Sparse Deep Learning: Theory and Computation](https://mlsanity.com/paper/2102.13229/) (2021)
- [Uncertainty Quantification for Sparse Deep Learning](https://mlsanity.com/paper/2002.11815/) (2020)
- [Variable selection for BART: An application to gene regulation](https://mlsanity.com/paper/1310.4887/) (2013)

## Built on by (most cited first, 7 of 7)

- [Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants](https://mlsanity.com/paper/2211.10575/) (2022)
- [Investigating Calibration and Corruption Robustness of Post-hoc Pruned Perception CNNs: An Image Classification Benchmark Study](https://mlsanity.com/paper/2405.20876/) (2024)
- [Sparse Deep Learning for Time Series Data: Theory and Applications](https://mlsanity.com/paper/2310.03243/) (2023)
- [Causal-StoNet: Causal Inference for High-Dimensional Complex Data](https://mlsanity.com/paper/2403.18994/) (2024)
- [Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks](https://mlsanity.com/paper/2206.05643/) (2022)
- [Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Prior](https://mlsanity.com/paper/2411.00969/) (2024)
- [Uncertainty Quantification for Large-Scale Deep Networks via Post-StoNet Modeling](https://mlsanity.com/paper/2508.01217/) (2025)

## Abstract (arXiv)

Deep learning has powered recent successes of artificial intelligence (AI). However, the deep neural network, as the basic model of deep learning, has suffered from issues such as local traps and miscalibration. In this paper, we provide a new framework for sparse deep learning, which has the above issues addressed in a coherent way. In particular, we lay down a theoretical foundation for sparse deep learning and propose prior annealing algorithms for learning sparse neural networks. The former has successfully tamed the sparse deep neural network into the framework of statistical modeling, enabling prediction uncertainty correctly quantified. The latter can be asymptotically guaranteed to converge to the global optimum, enabling the validity of the down-stream statistical inference. Numerical result indicates the superiority of the proposed method compared to the existing ones.