One Simple Trick to Fix Your Bayesian Neural Network
Not on the map. Why?
Abstract (arXiv)
One of the most popular estimation methods in Bayesian neural networks (BNN) is mean-field variational inference (MFVI). In this work, we show that neural networks with ReLU activation function induce posteriors, that are hard to fit with MFVI. We provide a theoretical justification for this phenomenon, study it empirically, and report the results of a series of experiments to investigate the effect of activation function on the calibration of BNNs. We find that using Leaky ReLU activations leads to more Gaussian-like weight posteriors and achieves a lower expected calibration error (ECE) than its ReLU-based counterpart.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 8 of its 12 references that are in the map. First posted on arXiv 26 Jul 2022.
Builds on
8 of 12 references.
References · 8
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2016Variational Inference: A Review for Statisticianscited by 1,472
- 2018Pyro: Deep Universal Probabilistic Programmingcited by 380
- 2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 280
- 2019A General Framework for Uncertainty Estimation in Deep Learningcited by 119
Built on by
Citing papers per year: 2025: 1.
1 paper.