# Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayes

arXiv: https://arxiv.org/abs/1906.05323 · HTML: https://mlsanity.com/paper/1906.05323/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 30 of 32)

- [Evidential Deep Learning for Open Set Action Recognition](https://mlsanity.com/paper/2107.10161/) (2021)
- [BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models](https://mlsanity.com/paper/2406.11675/) (2024)
- [Image-based Treatment Effect Heterogeneity](https://mlsanity.com/paper/2206.06417/) (2022)
- [SOAR: Scene-debiasing Open-set Action Recognition](https://mlsanity.com/paper/2309.01265/) (2023)
- [Bayesian Neural Networks Avoid Encoding Complex and Perturbation-Sensitive Concepts](https://mlsanity.com/paper/2302.13095/) (2023)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Domain Generalization with Small Data](https://mlsanity.com/paper/2402.06150/) (2024)
- [Implicit Variational Inference for High-Dimensional Posteriors](https://mlsanity.com/paper/2310.06643/) (2023)
- [An Empirical Bayes Framework for Open-Domain Dialogue Generation](https://mlsanity.com/paper/2311.10945/) (2023)
- [Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling](https://mlsanity.com/paper/2403.07282/) (2024)
- [Calibrating Bayesian Generative Machine Learning for Bayesiamplification](https://mlsanity.com/paper/2408.00838/) (2024)
- [Enhancing Trust in Large Language Models via Uncertainty-Calibrated Fine-Tuning](https://mlsanity.com/paper/2412.02904/) (2024)
- [EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimization](https://mlsanity.com/paper/2602.05165/) (2026)
- [Unsupervised Domain Adaptation for Low-dose CT Reconstruction via Bayesian Uncertainty Alignment](https://mlsanity.com/paper/2302.13251/) (2023)
- [Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks](https://mlsanity.com/paper/2307.07753/) (2023)
- [Bayesian Neural Networks for One-to-Many Mapping in Image Enhancement](https://mlsanity.com/paper/2501.14265/) (2025)
- [VLG: General Video Recognition with Web Textual Knowledge](https://mlsanity.com/paper/2212.01638/) (2022)
- [Robust Contrastive Active Learning with Feature-guided Query Strategies](https://mlsanity.com/paper/2109.06873/) (2021)
- [Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer](https://mlsanity.com/paper/2112.06281/) (2021)
- [Bayesian Domain Invariant Learning via Posterior Generalization of Parameter Distributions](https://mlsanity.com/paper/2310.16277/) (2023)
- [VI-PANN: Harnessing Transfer Learning and Uncertainty-Aware Variational Inference for Improved Generalization in Audio Pattern Recognition](https://mlsanity.com/paper/2401.05531/) (2024)
- [Flat Posterior Does Matter For Bayesian Model Averaging](https://mlsanity.com/paper/2406.15664/) (2024)
- [Is It Certainly a Deepfake? Reliability Analysis in Detection & Generation Ecosystem](https://mlsanity.com/paper/2509.17550/) (2025)
- [Piecewise Deterministic Markov Processes for Bayesian Neural Networks](https://mlsanity.com/paper/2302.08724/) (2023)
- [Variational Boosted Soft Trees](https://mlsanity.com/paper/2302.10706/) (2023)
- [Be Bayesian by Attachments to Catch More Uncertainty](https://mlsanity.com/paper/2310.13027/) (2023)
- [A Priori Uncertainty Quantification of Reacting Turbulence Closure Models using Bayesian Neural Networks](https://mlsanity.com/paper/2402.18729/) (2024)
- [Logit Disagreement: OoD Detection with Bayesian Neural Networks](https://mlsanity.com/paper/2502.15648/) (2025)
- [Bayesian Deep Learning for Convective Initiation Nowcasting Uncertainty Estimation](https://mlsanity.com/paper/2507.16219/) (2025)
- [PEdger++: Practical Edge Detection via Assembling Cross Information](https://mlsanity.com/paper/2508.11961/) (2025)

## Abstract (arXiv)

Stochastic variational inference for Bayesian deep neural network (DNN) requires specifying priors and approximate posterior distributions over neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dimensional weight space. We propose MOdel Priors with Empirical Bayes using DNN (MOPED) method to choose informed weight priors in Bayesian neural networks. We formulate a two-stage hierarchical modeling, first find the maximum likelihood estimates of weights with DNN, and then set the weight priors using empirical Bayes approach to infer the posterior with variational inference. We empirically evaluate the proposed approach on real-world tasks including image classification, video activity recognition and audio classification with varying complex neural network architectures. We also evaluate our proposed approach on diabetic retinopathy diagnosis task and benchmark with the state-of-the-art Bayesian deep learning techniques. We demonstrate MOPED method enables scalable variational inference and provides reliable uncertainty quantification.