# Model Architecture Adaption for Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2202.04392 · HTML: https://mlsanity.com/paper/2202.04392/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours](https://mlsanity.com/paper/1904.02877/) (2019)
- [Bayesian Deep Learning via Subnetwork Inference](https://mlsanity.com/paper/2010.14689/) (2020)
- [BayesNAS: A Bayesian Approach for Neural Architecture Search](https://mlsanity.com/paper/1905.04919/) (2019)
- [Learned Low Precision Graph Neural Networks](https://mlsanity.com/paper/2009.09232/) (2020)
- [On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation](https://mlsanity.com/paper/2001.08049/) (2020)
- [Probabilistic Neural Architecture Search](https://mlsanity.com/paper/1902.05116/) (2019)
- [Probabilistic Dual Network Architecture Search on Graphs](https://mlsanity.com/paper/2003.09676/) (2020)
- [The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning](https://mlsanity.com/paper/1811.12535/) (2018)
- [Rapid Model Architecture Adaption for Meta-Learning](https://mlsanity.com/paper/2109.04925/) (2021)

## Abstract (arXiv)

Bayesian Neural Networks (BNNs) offer a mathematically grounded framework to quantify the uncertainty of model predictions but come with a prohibitive computation cost for both training and inference. In this work, we show a novel network architecture search (NAS) that optimizes BNNs for both accuracy and uncertainty while having a reduced inference latency. Different from canonical NAS that optimizes solely for in-distribution likelihood, the proposed scheme searches for the uncertainty performance using both in- and out-of-distribution data. Our method is able to search for the correct placement of Bayesian layer(s) in a network. In our experiments, the searched models show comparable uncertainty quantification ability and accuracy compared to the state-of-the-art (deep ensemble). In addition, the searched models use only a fraction of the runtime compared to many popular BNN baselines, reducing the inference runtime cost by 2.98 × and 2.92 × respectively on the CIFAR10 dataset when compared to MCDropout and deep ensemble.