# Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks

arXiv: https://arxiv.org/abs/2005.08110 · HTML: https://mlsanity.com/paper/2005.08110/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2020.

## Built on by (most cited first, 14 of 14)

- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Generative Transformer for Accurate and Reliable Salient Object Detection](https://mlsanity.com/paper/2104.10127/) (2021)
- [Distill n' Explain: explaining graph neural networks using simple surrogates](https://mlsanity.com/paper/2303.10139/) (2023)
- [Federated Learning with Uncertainty via Distilled Predictive Distributions](https://mlsanity.com/paper/2206.07562/) (2022)
- [Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts](https://mlsanity.com/paper/2302.06495/) (2023)
- [Self-Distribution Distillation: Efficient Uncertainty Estimation](https://mlsanity.com/paper/2203.08295/) (2022)
- [Post-hoc loss-calibration for Bayesian neural networks](https://mlsanity.com/paper/2106.06997/) (2021)
- [Dense Uncertainty Estimation](https://mlsanity.com/paper/2110.06427/) (2021)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [Impact of Parameter Sparsity on Stochastic Gradient MCMC Methods for Bayesian Deep Learning](https://mlsanity.com/paper/2202.03770/) (2022)
- [Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation](https://mlsanity.com/paper/2307.09929/) (2023)
- [Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors](https://mlsanity.com/paper/2605.27967/) (2026)
- [Challenges and Opportunities in Approximate Bayesian Deep Learning for Intelligent IoT Systems](https://mlsanity.com/paper/2112.01675/) (2021)
- [A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions](https://mlsanity.com/paper/2609.33727/) (2026)

## Abstract (arXiv)

In this paper, we present a general framework for distilling expectations with respect to the Bayesian posterior distribution of a deep neural network classifier, extending prior work on the Bayesian Dark Knowledge framework. The proposed framework takes as input "teacher" and student model architectures and a general posterior expectation of interest. The distillation method performs an online compression of the selected posterior expectation using iteratively generated Monte Carlo samples. We focus on the posterior predictive distribution and expected entropy as distillation targets. We investigate several aspects of this framework including the impact of uncertainty and the choice of student model architecture. We study methods for student model architecture search from a speed-storage-accuracy perspective and evaluate down-stream tasks leveraging entropy distillation including uncertainty ranking and out-of-distribution detection.