Masked Bayesian Neural Networks : Computation and Optimality
Not on the map. Why?
Abstract (arXiv)
As data size and computing power increase, the architectures of deep neural networks (DNNs) have been getting more complex and huge, and thus there is a growing need to simplify such complex and huge DNNs. In this paper, we propose a novel sparse Bayesian neural network (BNN) which searches a good DNN with an appropriate complexity. We employ the masking variables at each node which can turn off some nodes according to the posterior distribution to yield a nodewise sparse DNN. We devise a prior distribution such that the posterior distribution has theoretical optimalities (i.e. minimax optimality and adaptiveness), and develop an efficient MCMC algorithm. By analyzing several benchmark datasets, we illustrate that the proposed BNN performs well compared to other existing methods in the sense that it discovers well condensed DNN architectures with similar prediction accuracy and uncertainty quantification compared to large DNNs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 41 of its 69 references that are in the map. First posted on arXiv 2 Jun 2022.
Builds on
41 of 69 references.
References · 41
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2016Variational Inference: A Review for Statisticianscited by 1,466
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
- 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
- 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
- 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
29 more
- 2015Variational Dropout and the Local Reparameterization Trickcited by 489
- 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
- 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
- 2019Verified Uncertainty Calibrationcited by 307
- 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
- 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
- 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
- 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
- 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
- 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
- 2020Uncertainty Quantification and Deep Ensemblescited by 138
- 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
- 2017Bayesian Compression for Deep Learningcited by 116
- 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
- 2014Collaborative Deep Learning for Recommender Systemscited by 95
- 2021Oops I Took A Gradient: Scalable Sampling for Discrete Distributionscited by 92
- 2019A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Taskscited by 62
- 2019Quality of Uncertainty Quantification for Bayesian Neural Network Inferencecited by 58
- 2017Model Selection in Bayesian Neural Networks via Horseshoe Priorscited by 52
- 2018Posterior Concentration for Sparse Deep Learningcited by 50
- 2014Bayesian linear regression with sparse priorscited by 40
- 2020Efficient Variational Inference for Sparse Deep Learning with Theoretical Guaranteecited by 37
- 2019Convergence Rates of Variational Inference in Sparse Deep Learningcited by 27
- 2019An Adaptive Empirical Bayesian Method for Sparse Deep Learningcited by 20
- 2019Bayesian Inference for Large Scale Image Classificationcited by 18
- 2021Adaptive variational Bayes: Optimality, computation and applicationscited by 17
- 2021A Bayesian neural network predicts the dissolution of compact planetary systemscited by 15
- 2020Bayesian Sparse learning with preconditioned stochastic gradient MCMC and its applicationscited by 6
Built on by
Citing papers per year: 2023: 1.
1 paper.