mlsanityData to 7 Oct 2026

Bayesian Dark Knowledge

arXiv 1506.04416 ↗First posted 14 Jun 2015cs.LG stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

We consider the problem of Bayesian parameter estimation for deep neural networks, which is important in problem settings where we may have little data, and/ or where we need accurate posterior predictive densities, e.g., for applications involving bandits or active learning. One simple approach to this is to use online Monte Carlo methods, such as SGLD (stochastic gradient Langevin dynamics). Unfortunately, such a method needs to store many copies of the parameters (which wastes memory), and needs to make predictions using many versions of the model (which wastes time). We describe a method for "distilling" a Monte Carlo approximation to the posterior predictive density into a more compact form, namely a single deep neural network. We compare to two very recent approaches to Bayesian neural networks, namely an approach based on expectation propagation [Hernandez-Lobato and Adams, 2015] and an approach based on variational Bayes [Blundell et al., 2015]. Our method performs better than both of these, is much simpler to implement, and uses less computation at test time.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 15; 2022: 12; 2023: 7; 2024: 2; 2025: 5; 2026: 3.

45 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  4. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
  5. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  6. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  7. 2021The Bayesian Learning Rulecited by 95
  8. 2022Blueprint Separable Residual Network for Efficient Image Super-Resolutioncited by 44
  9. 2021LiBRe: A Practical Bayesian Approach to Adversarial Detectioncited by 40
  10. 2021Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equationscited by 38
  11. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  12. 2022STUN: Self-Teaching Uncertainty Estimation for Place Recognitioncited by 13
33 more
  1. 2022Low-Precision Stochastic Gradient Langevin Dynamicscited by 12
  2. 2021Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaptationcited by 11
  3. 2022Federated Learning with Uncertainty via Distilled Predictive Distributionscited by 11
  4. 2022Scalable Bayesian Uncertainty Quantification for Neural Network Potentials: Promise and Pitfallscited by 11
  5. 2021Post-hoc loss-calibration for Bayesian neural networkscited by 9
  6. 2023Fully Bayesian Autoencoders with Latent Sparse Gaussian Processescited by 9
  7. 2022Task Agnostic and Post-hoc Unseen Distribution Detectioncited by 8
  8. 2023Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metricscited by 8
  9. 2023Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detectioncited by 7
  10. 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
  11. 2021Introspective Robot Perception using Smoothed Predictions from Bayesian Neural Networkscited by 5
  12. 2025BARNN: A Bayesian Autoregressive and Recurrent Neural Networkcited by 5
  13. 2023Exploration via Epistemic Value Estimationcited by 4
  14. 2021The Bayesian Method of Tensor Networkscited by 3
  15. 2022Functional Ensemble Distillationcited by 3
  16. 2025Large Language Models for Bioinformaticscited by 3
  17. 2022Impact of Parameter Sparsity on Stochastic Gradient MCMC Methods for Bayesian Deep Learningcited by 2
  18. 2023Variational Predictioncited by 2
  19. 2024Model and Feature Diversity for Bayesian Neural Networks in Mutual Learningcited by 2
  20. 2020Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learningcited by 1
  21. 2021Accurate and Reliable Forecasting using Stochastic Differential Equationscited by 1
  22. 2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Modelscited by 1
  23. 2026Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priorscited by 1
  24. 2021Risk factor identification for incident heart failure using neural network distillation and variable selectioncited by 0
  25. 2021Generative Particle Variational Inference via Estimation of Functional Gradientscited by 0
  26. 2021Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systemscited by 0
  27. 2021Challenges and Opportunities in Approximate Bayesian Deep Learning for Intelligent IoT Systemscited by 0
  28. 2022SBPF: Sensitiveness Based Pruning Framework For Convolutional Neural Network On Image Classificationcited by 0
  29. 2023Analyzing Compression Techniques for Computer Visioncited by 0
  30. 2025GraphPPD: Posterior Predictive Modelling for Graph-Level Inferencecited by 0
  31. 2025Probabilistic Machine Learning for Uncertainty-Aware Diagnosis of Industrial Systemscited by 0
  32. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
  33. 2026A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensionscited by 0