# Adversarial Distillation of Bayesian Neural Network Posteriors

arXiv: https://arxiv.org/abs/1806.10317 · HTML: https://mlsanity.com/paper/1806.10317/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2018.

## Built on by (most cited first, 27 of 27)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights](https://mlsanity.com/paper/2209.14733/) (2022)
- [Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations](https://mlsanity.com/paper/2102.06559/) (2021)
- [Interpreting the Weight Space of Customized Diffusion Models](https://mlsanity.com/paper/2406.09413/) (2024)
- [PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics](https://mlsanity.com/paper/2106.02993/) (2021)
- [Fairness through Aleatoric Uncertainty](https://mlsanity.com/paper/2304.03646/) (2023)
- [Federated Learning with Uncertainty via Distilled Predictive Distributions](https://mlsanity.com/paper/2206.07562/) (2022)
- [Low-Precision Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2206.09909/) (2022)
- [Dense Uncertainty Estimation](https://mlsanity.com/paper/2110.06427/) (2021)
- [Meta-Learning via Classifier(-free) Diffusion Guidance](https://mlsanity.com/paper/2210.08942/) (2022)
- [Fully Bayesian Autoencoders with Latent Sparse Gaussian Processes](https://mlsanity.com/paper/2302.04534/) (2023)
- [Confidence estimation of classification based on the distribution of the neural network output layer](https://mlsanity.com/paper/2210.07745/) (2022)
- [Introspective Robot Perception using Smoothed Predictions from Bayesian Neural Networks](https://mlsanity.com/paper/2109.12869/) (2021)
- [PriMask: Cascadable and Collusion-Resilient Data Masking for Mobile Cloud Inference](https://mlsanity.com/paper/2211.06716/) (2022)
- [Hyp-UML: Hyperbolic Image Retrieval with Uncertainty-aware Metric Learning](https://mlsanity.com/paper/2310.08390/) (2023)
- [Error-Driven Uncertainty Aware Training](https://mlsanity.com/paper/2405.01205/) (2024)
- [PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection](https://mlsanity.com/paper/2410.20631/) (2024)
- [Efficient Bayes Inference in Neural Networks through Adaptive Importance Sampling](https://mlsanity.com/paper/2210.00993/) (2022)
- [Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning](https://mlsanity.com/paper/2407.02721/) (2024)
- [Recall-Oriented Continual Learning with Generative Adversarial Meta-Model](https://mlsanity.com/paper/2403.03082/) (2024)
- [Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors](https://mlsanity.com/paper/2605.27967/) (2026)
- [Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges](https://mlsanity.com/paper/2107.02894/) (2021)
- [Challenges and Opportunities in Approximate Bayesian Deep Learning for Intelligent IoT Systems](https://mlsanity.com/paper/2112.01675/) (2021)
- [Bayesian Neural Network Inference via Implicit Models and the Posterior Predictive Distribution](https://mlsanity.com/paper/2209.02188/) (2022)
- [Non-Generative Energy Based Models](https://mlsanity.com/paper/2304.01297/) (2023)
- [Hyper-Representations: Learning from Populations of Neural Networks](https://mlsanity.com/paper/2410.05107/) (2024)

## Abstract (arXiv)

Bayesian neural networks (BNNs) allow us to reason about uncertainty in a principled way. Stochastic Gradient Langevin Dynamics (SGLD) enables efficient BNN learning by drawing samples from the BNN posterior using mini-batches. However, SGLD and its extensions require storage of many copies of the model parameters, a potentially prohibitive cost, especially for large neural networks. We propose a framework, Adversarial Posterior Distillation, to distill the SGLD samples using a Generative Adversarial Network (GAN). At test-time, samples are generated by the GAN. We show that this distillation framework incurs no loss in performance on recent BNN applications including anomaly detection, active learning, and defense against adversarial attacks. By construction, our framework not only distills the Bayesian predictive distribution, but the posterior itself. This allows one to compute quantities such as the approximate model variance, which is useful in downstream tasks. To our knowledge, these are the first results applying MCMC-based BNNs to the aforementioned downstream applications.