# Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

arXiv: https://arxiv.org/abs/1603.04733 · HTML: https://mlsanity.com/paper/1603.04733/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2016.

## Built on by (most cited first, 30 of 82)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06329/) (2021)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration](https://mlsanity.com/paper/2111.15430/) (2021)
- [Inconsistency-aware Uncertainty Estimation for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2110.08762/) (2021)
- [Quantifying Uncertainty in Deep Spatiotemporal Forecasting](https://mlsanity.com/paper/2105.11982/) (2021)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [LiBRe: A Practical Bayesian Approach to Adversarial Detection](https://mlsanity.com/paper/2103.14835/) (2021)
- [BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inference](https://mlsanity.com/paper/2310.11142/) (2023)
- [Accelerated Linearized Laplace Approximation for Bayesian Deep Learning](https://mlsanity.com/paper/2210.12642/) (2022)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [Robustness via Cross-Domain Ensembles](https://mlsanity.com/paper/2103.10919/) (2021)
- [The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective](https://mlsanity.com/paper/2106.06529/) (2021)
- [Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2405.03425/) (2024)
- [BayesFormer: Transformer with Uncertainty Estimation](https://mlsanity.com/paper/2206.00826/) (2022)
- [Uncertainty-Aware Distillation for Semi-Supervised Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2301.09964/) (2023)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [Multiclass Confidence and Localization Calibration for Object Detection](https://mlsanity.com/paper/2306.08271/) (2023)
- [A Review of Pseudo-Labeling for Computer Vision](https://mlsanity.com/paper/2408.07221/) (2024)
- [Improving Neural Additive Models with Bayesian Principles](https://mlsanity.com/paper/2305.16905/) (2023)
- [Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks](https://mlsanity.com/paper/2205.10041/) (2022)
- [Pathologies in priors and inference for Bayesian transformers](https://mlsanity.com/paper/2110.04020/) (2021)
- [DeepGLEAM: A hybrid mechanistic and deep learning model for COVID-19 forecasting](https://mlsanity.com/paper/2102.06684/) (2021)
- [Towards Accelerated Model Training via Bayesian Data Selection](https://mlsanity.com/paper/2308.10544/) (2023)
- [Cal-DETR: Calibrated Detection Transformer](https://mlsanity.com/paper/2311.03570/) (2023)
- [Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization](https://mlsanity.com/paper/2304.08309/) (2023)

## Abstract (arXiv)

We introduce a variational Bayesian neural network where the parameters are governed via a probability distribution on random matrices. Specifically, we employ a matrix variate Gaussian citegupta1999matrix parameter posterior distribution where we explicitly model the covariance among the input and output dimensions of each layer. Furthermore, with approximate covariance matrices we can achieve a more efficient way to represent those correlations that is also cheaper than fully factorized parameter posteriors. We further show that with the "local reprarametrization trick" citekingma2015variational on this posterior distribution we arrive at a Gaussian Process citerasmussen2006gaussian interpretation of the hidden units in each layer and we, similarly with citegal2015dropout, provide connections with deep Gaussian processes. We continue in taking advantage of this duality and incorporate "pseudo-data" citesnelson2005sparse in our model, which in turn allows for more efficient sampling while maintaining the properties of the original model. The validity of the proposed approach is verified through extensive experiments.