# An Adaptive Empirical Bayesian Method for Sparse Deep Learning

arXiv: https://arxiv.org/abs/1910.10791 · HTML: https://mlsanity.com/paper/1910.10791/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2019.

## Built on by (most cited first, 20 of 20)

- [Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants](https://mlsanity.com/paper/2211.10575/) (2022)
- [Information Directed Sampling for Sparse Linear Bandits](https://mlsanity.com/paper/2105.14267/) (2021)
- [Sparse Uncertainty Representation in Deep Learning with Inducing Weights](https://mlsanity.com/paper/2105.14594/) (2021)
- [Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network](https://mlsanity.com/paper/2210.04349/) (2022)
- [Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference](https://mlsanity.com/paper/2305.14765/) (2023)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [Mixed Effects Neural ODE: A Variational Approximation for Analyzing the Dynamics of Panel Data](https://mlsanity.com/paper/2202.09463/) (2022)
- [Causal-StoNet: Causal Inference for High-Dimensional Complex Data](https://mlsanity.com/paper/2403.18994/) (2024)
- [AutoAttention: Automatic Field Pair Selection for Attention in User Behavior Modeling](https://mlsanity.com/paper/2210.15154/) (2022)
- [Sparsifying Bayesian neural networks with latent binary variables and normalizing flows](https://mlsanity.com/paper/2305.03395/) (2023)
- [Active Learning with Bayesian Multi-Fidelity Laplace Neural Operators for Oscillatory Parametric PDEs](https://mlsanity.com/paper/2502.00550/) (2025)
- [Uncertainty Quantification for Physics-Informed Neural Networks with Extended Fiducial Inference](https://mlsanity.com/paper/2505.19136/) (2025)
- [Masked Bayesian Neural Networks : Computation and Optimality](https://mlsanity.com/paper/2206.00853/) (2022)
- [Incorporating functional summary information in Bayesian neural networks using a Dirichlet process likelihood approach](https://mlsanity.com/paper/2207.01234/) (2022)
- [Extended Fiducial Inference for Individual Treatment Effects via Deep Neural Networks](https://mlsanity.com/paper/2505.01995/) (2025)
- [A New Paradigm for Generative Adversarial Networks based on Randomized Decision Rules](https://mlsanity.com/paper/2306.13641/) (2023)
- [Variational Sampling of Temporal Trajectories](https://mlsanity.com/paper/2403.11418/) (2024)
- [Uncertainty Quantification for Large-Scale Deep Networks via Post-StoNet Modeling](https://mlsanity.com/paper/2508.01217/) (2025)
- [SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions](https://mlsanity.com/paper/2510.08999/) (2025)
- [Sparsity via Hyperpriors: A Theoretical and Algorithmic Study under Empirical Bayes Framework](https://mlsanity.com/paper/2511.06235/) (2025)

## Abstract (arXiv)

We propose a novel adaptive empirical Bayesian method for sparse deep learning, where the sparsity is ensured via a class of self-adaptive spike-and-slab priors. The proposed method works by alternatively sampling from an adaptive hierarchical posterior distribution using stochastic gradient Markov Chain Monte Carlo (MCMC) and smoothly optimizing the hyperparameters using stochastic approximation (SA). We further prove the convergence of the proposed method to the asymptotically correct distribution under mild conditions. Empirical applications of the proposed method lead to the state-of-the-art performance on MNIST and Fashion MNIST with shallow convolutional neural networks and the state-of-the-art compression performance on CIFAR10 with Residual Networks. The proposed method also improves resistance to adversarial attacks.