# Analytic Mutual Information in Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2201.09815 · HTML: https://mlsanity.com/paper/2201.09815/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 54 references that are in the map.
- First posted on arXiv 24 Jan 2022.

## Builds on (most cited first, 13 of 13)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Uncertainty in Neural Networks: Approximately Bayesian Ensembling](https://mlsanity.com/paper/1810.05546/) (2018)
- [Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification](https://mlsanity.com/paper/1805.10915/) (2018)
- [Fast Predictive Uncertainty for Classification with Bayesian Deep Networks](https://mlsanity.com/paper/2003.01227/) (2020)
- [Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle](https://mlsanity.com/paper/2105.14559/) (2021)

## Built on by (most cited first, 5 of 5)

- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation](https://mlsanity.com/paper/2405.06424/) (2024)
- [Bayesian Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2408.01694/) (2024)
- [Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty](https://mlsanity.com/paper/2606.17312/) (2026)
- [Bayesian Topological Convolutional Neural Nets](https://mlsanity.com/paper/2510.11704/) (2025)

## Abstract (arXiv)

Bayesian neural networks have successfully designed and optimized a robust neural network model in many application problems, including uncertainty quantification. However, with its recent success, information-theoretic understanding about the Bayesian neural network is still at an early stage. Mutual information is an example of an uncertainty measure in a Bayesian neural network to quantify epistemic uncertainty. Still, no analytic formula is known to describe it, one of the fundamental information measures to understand the Bayesian deep learning framework. In this paper, we derive the analytical formula of the mutual information between model parameters and the predictive output by leveraging the notion of the point process entropy. Then, as an application, we discuss the parameter estimation of the Dirichlet distribution and show its practical application in the active learning uncertainty measures by demonstrating that our analytical formula can improve the performance of active learning further in practice.