mlsanityData to 14 Oct 2026

An Adaptive Empirical Bayesian Method for Sparse Deep Learning

arXiv 1910.10791 ↗First posted 23 Oct 2019stat.ML cs.LG stat.MEcited by 20

Not on the map. Why?

Abstract (arXiv)

We propose a novel adaptive empirical Bayesian method for sparse deep learning, where the sparsity is ensured via a class of self-adaptive spike-and-slab priors. The proposed method works by alternatively sampling from an adaptive hierarchical posterior distribution using stochastic gradient Markov Chain Monte Carlo (MCMC) and smoothly optimizing the hyperparameters using stochastic approximation (SA). We further prove the convergence of the proposed method to the asymptotically correct distribution under mild conditions. Empirical applications of the proposed method lead to the state-of-the-art performance on MNIST and Fashion MNIST with shallow convolutional neural networks and the state-of-the-art compression performance on CIFAR10 with Residual Networks. The proposed method also improves resistance to adversarial attacks.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 6; 2023: 3; 2024: 3; 2025: 6.

20 papers.

Sort:

  1. 2022Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constantscited by 24
  2. 2021Information Directed Sampling for Sparse Linear Banditscited by 20
  3. 2021Sparse Uncertainty Representation in Deep Learning with Inducing Weightscited by 16
  4. 2022Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Networkcited by 10
  5. 2023Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inferencecited by 9
  6. 2024Training Bayesian Neural Networks with Sparse Subspace Variational Inferencecited by 7
  7. 2022Mixed Effects Neural ODE: A Variational Approximation for Analyzing the Dynamics of Panel Datacited by 3
  8. 2024Causal-StoNet: Causal Inference for High-Dimensional Complex Datacited by 3
  9. 2022AutoAttention: Automatic Field Pair Selection for Attention in User Behavior Modelingcited by 2
  10. 2023Sparsifying Bayesian neural networks with latent binary variables and normalizing flowscited by 2
  11. 2025Active Learning with Bayesian Multi-Fidelity Laplace Neural Operators for Oscillatory Parametric PDEscited by 2
  12. 2025Uncertainty Quantification for Physics-Informed Neural Networks with Extended Fiducial Inferencecited by 2
8 more
  1. 2022Masked Bayesian Neural Networks : Computation and Optimalitycited by 1
  2. 2022Incorporating functional summary information in Bayesian neural networks using a Dirichlet process likelihood approachcited by 1
  3. 2025Extended Fiducial Inference for Individual Treatment Effects via Deep Neural Networkscited by 1
  4. 2023A New Paradigm for Generative Adversarial Networks based on Randomized Decision Rulescited by 0
  5. 2024Variational Sampling of Temporal Trajectoriescited by 0
  6. 2025Uncertainty Quantification for Large-Scale Deep Networks via Post-StoNet Modelingcited by 0
  7. 2025SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributionscited by 0
  8. 2025Sparsity via Hyperpriors: A Theoretical and Algorithmic Study under Empirical Bayes Frameworkcited by 0