mlsanityData to 13 Oct 2026

Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer

arXiv 2112.06281 ↗First posted 12 Dec 2021cs.LG cs.AI stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Bayesian neural networks (BNNs) have become a principal approach to alleviate overconfident predictions in deep learning, but they often suffer from scaling issues due to a large number of distribution parameters. In this paper, we discover that the first layer of a deep network possesses multiple disparate optima when solely retrained. This indicates a large posterior variance when the first layer is altered by a Bayesian layer, which motivates us to design a spatial-temporal-fusion BNN (STF-BNN) for efficiently scaling BNNs to large models: (1) first normally train a neural network from scratch to realize fast training; and (2) the first layer is converted to Bayesian and inferred by employing stochastic variational inference, while other layers are fixed. Compared to vanilla BNNs, our approach can greatly reduce the training time and the number of parameters, which contributes to scale BNNs efficiently. We further provide theoretical guarantees on the generalizability and the capability of mitigating overconfidence of STF-BNN. Comprehensive experiments demonstrate that STF-BNN (1) achieves the state-of-the-art performance on prediction and uncertainty quantification; (2) significantly improves adversarial robustness and privacy preservation; and (3) considerably reduces training time and memory costs.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 43 of its 72 references that are in the map. First posted on arXiv 12 Dec 2021.

Builds on

43 of 72 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,336
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
  5. 2017On Calibration of Modern Neural Networkscited by 4,902
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,430
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,328
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,902
  9. 2016Wide Residual Networkscited by 3,063
  10. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,992
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,862
  12. 2016Deep Learning with Differential Privacycited by 2,718
31 more
  1. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,273
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,511
  3. 2016Variational Inference: A Review for Statisticianscited by 1,469
  4. 2017Renyi Differential Privacycited by 648
  5. 2019Adversarial Training for Free!cited by 641
  6. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  7. 2014On the Number of Linear Regions of Deep Neural Networkscited by 380
  8. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  9. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 351
  10. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  11. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  12. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 299
  13. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 229
  14. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  15. 2020Geometry-aware Instance-reweighted Adversarial Trainingcited by 198
  16. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  17. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  18. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  19. 2019Privacy Risks of Securing Machine Learning Models against Adversarial Examplescited by 141
  20. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  21. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  22. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  23. 2020SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimatescited by 75
  24. 2018Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Networkcited by 69
  25. 2020Recent advances in deep learning theorycited by 41
  26. 2019Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learningcited by 37
  27. 2015An Empirical Evaluation of Deep Learning on Highway Drivingcited by 35
  28. 2019Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayescited by 32
  29. 2020An Uncertainty-aware Transfer Learning-based Framework for Covid-19 Diagnosiscited by 22
  30. 2020Tighter Generalization Bounds for Iterative Differentially Private Learning Algorithmscited by 14
  31. 2019Evaluating and Boosting Uncertainty Quantification in Classificationcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63