# Understanding Approximation for Bayesian Inference in Neural Networks

arXiv: https://arxiv.org/abs/2211.06139 · HTML: https://mlsanity.com/paper/2211.06139/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 159 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Cost-Effective Active Learning for Deep Image Classification](https://mlsanity.com/paper/1701.03551/) (2017)
- [Noisy Natural Gradient as Variational Inference](https://mlsanity.com/paper/1712.02390/) (2017)
- [Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks](https://mlsanity.com/paper/1911.09737/) (2019)
- [Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors](https://mlsanity.com/paper/2205.10279/) (2022)
- [SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient](https://mlsanity.com/paper/1811.04504/) (2018)
- [CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions](https://mlsanity.com/paper/2009.09929/) (2020)
- [Half a Percent of Labels is Enough: Efficient Animal Detection in UAV Imagery using Deep CNNs and Active Learning](https://mlsanity.com/paper/1907.07319/) (2019)
- [Cost-Effective Training of Deep CNNs with Active Model Adaptation](https://mlsanity.com/paper/1802.05394/) (2018)

## Built on by (most cited first, 1 of 1)

- [Making Better Use of Unlabelled Data in Bayesian Active Learning](https://mlsanity.com/paper/2404.17249/) (2024)

## Abstract (arXiv)

Bayesian inference has theoretical attractions as a principled framework for reasoning about beliefs. However, the motivations of Bayesian inference which claim it to be the only 'rational' kind of reasoning do not apply in practice. They create a binary split in which all approximate inference is equally 'irrational'. Instead, we should ask ourselves how to define a spectrum of more- and less-rational reasoning that explains why we might prefer one Bayesian approximation to another. I explore approximate inference in Bayesian neural networks and consider the unintended interactions between the probabilistic model, approximating distribution, optimization algorithm, and dataset. The complexity of these interactions highlights the difficulty of any strategy for evaluating Bayesian approximations which focuses entirely on the method, outside the context of specific datasets and decision-problems. For given applications, the expected utility of the approximate posterior can measure inference quality. To assess a model's ability to incorporate different parts of the Bayesian framework we can identify desirable characteristic behaviours of Bayesian reasoning and pick decision-problems that make heavy use of those behaviours. Here, we use continual learning (testing the ability to update sequentially) and active learning (testing the ability to represent credence). But existing continual and active learning set-ups pose challenges that have nothing to do with posterior quality which can distort their ability to evaluate Bayesian approximations. These unrelated challenges can be removed or reduced, allowing better evaluation of approximate inference methods.