mlsanityData to 7 Oct 2026

Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee

arXiv 2011.07439 ↗First posted 15 Nov 2020stat.ML cs.LG math.STcited by 37

Not on the map. Why?

Abstract (arXiv)

Sparse deep learning aims to address the challenge of huge storage consumption by deep neural networks, and to recover the sparse structure of target functions. Although tremendous empirical successes have been achieved, most sparse deep learning algorithms are lacking of theoretical support. On the other hand, another line of works have proposed theoretical frameworks that are computationally infeasible. In this paper, we train sparse deep neural networks with a fully Bayesian treatment under spike-and-slab priors, and develop a set of computationally efficient variational inferences via continuous relaxation of Bernoulli distribution. The variational posterior contraction rate is provided, which justifies the consistency of the proposed variational Bayes method. Notably, our empirical results demonstrate that this variational procedure provides uncertainty quantification in terms of Bayesian predictive distribution and is also capable to accomplish consistent variable selection by training a sparse multi-layer neural network.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 11; 2024: 10; 2025: 5; 2026: 2.

37 papers.

Sort:

  1. 2022Personalized Federated Learning via Variational Bayesian Inferencecited by 68
  2. 2022Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constantscited by 24
  3. 2022Efficient Stein Variational Inference for Reliable Distribution-lossless Network Pruningcited by 16
  4. 2021Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Detailscited by 14
  5. 2023Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inferencecited by 9
  6. 2024Posterior and variational inference for deep neural networks with heavy-tailed weightscited by 9
  7. 2023FedHB: Hierarchical Bayesian Federated Learningcited by 8
  8. 2021Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learningcited by 7
  9. 2024Training Bayesian Neural Networks with Sparse Subspace Variational Inferencecited by 7
  10. 2024Misclassification bounds for PAC-Bayesian sparse deep learningcited by 7
  11. 2022Sequential Bayesian Neural Subnetwork Ensemblescited by 5
  12. 2023Federated Learning via Variational Bayesian Inference: Personalization, Sparsity and Clusteringcited by 5
25 more
  1. 2023Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertaintycited by 5
  2. 2024Deep Horseshoe Gaussian Processescited by 5
  3. 2023Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networkscited by 4
  4. 2024FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantificationcited by 4
  5. 2022Mixed Effects Neural ODE: A Variational Approximation for Analyzing the Dynamics of Panel Datacited by 3
  6. 2022On the optimization and pruning for Bayesian deep learningcited by 2
  7. 2023Sparsifying Bayesian neural networks with latent binary variables and normalizing flowscited by 2
  8. 2023Bayesian Personalized Federated Learning with Shared and Personalized Uncertainty Representationscited by 2
  9. 2024BMRS: Bayesian Model Reduction for Structured Pruningcited by 2
  10. 2025Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionalitycited by 2
  11. 2022Masked Bayesian Neural Networks : Computation and Optimalitycited by 1
  12. 2023Projective Integral Updates for High-Dimensional Variational Inferencecited by 1
  13. 2023Personalized Federated Learning via Amortized Bayesian Meta-Learningcited by 1
  14. 2023Bayesian sparsification for deep neural networks with Bayesian model reductioncited by 1
  15. 2025Bayesian Computation in Deep Learningcited by 1
  16. 2025Explainable Bayesian deep learning through input-skip Latent Binary Bayesian Neural Networkscited by 1
  17. 2025Bayesian Coreset Optimization for Personalized Federated Learningcited by 1
  18. 2023A Hierarchical Bayesian Model for Deep Few-Shot Meta Learningcited by 0
  19. 2024Variational Sampling of Temporal Trajectoriescited by 0
  20. 2024Personalizing Low-Rank Bayesian Neural Networks Via Federated Learningcited by 0
  21. 2024Efficient Model Compression for Bayesian Neural Networkscited by 0
  22. 2024Variational Bayesian Bow tie Neural Networks with Shrinkagecited by 0
  23. 2025SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributionscited by 0
  24. 2026Posterior Concentration of Bayesian Physics-Informed Neural Networks for Elliptic PDEscited by 0
  25. 2026Knockoffs-based False Discovery Rate Control and Simplification for Deep Neural Networkscited by 0