# Walsh-Hadamard Variational Inference for Bayesian Deep Learning

arXiv: https://arxiv.org/abs/1905.11248 · HTML: https://mlsanity.com/paper/1905.11248/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 8 of 8)

- [Flat Seeking Bayesian Neural Networks](https://mlsanity.com/paper/2302.02713/) (2023)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Improving Generalization with Flat Hilbert Bayesian Inference](https://mlsanity.com/paper/2410.04196/) (2024)
- [On permutation symmetries in Bayesian neural network posteriors: a variational perspective](https://mlsanity.com/paper/2310.10171/) (2023)
- [ADC/DAC-Free Analog Acceleration of Deep Neural Networks with Frequency Transformation](https://mlsanity.com/paper/2309.01771/) (2023)
- [Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models](https://mlsanity.com/paper/2506.07247/) (2025)
- [Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning](https://mlsanity.com/paper/2407.02721/) (2024)
- [Containing Analog Data Deluge at Edge through Frequency-Domain Compression in Collaborative Compute-in-Memory Networks](https://mlsanity.com/paper/2309.11048/) (2023)

## Abstract (arXiv)

Over-parameterized models, such as DeepNets and ConvNets, form a class of models that are routinely adopted in a wide variety of applications, and for which Bayesian inference is desirable but extremely challenging. Variational inference offers the tools to tackle this challenge in a scalable way and with some degree of flexibility on the approximation, but for over-parameterized models this is challenging due to the over-regularization property of the variational objective. Inspired by the literature on kernel methods, and in particular on structured approximations of distributions of random matrices, this paper proposes Walsh-Hadamard Variational Inference (WHVI), which uses Walsh-Hadamard-based factorization strategies to reduce the parameterization and accelerate computations, thus avoiding over-regularization issues with the variational objective. Extensive theoretical and empirical analyses demonstrate that WHVI yields considerable speedups and model reductions compared to other techniques to carry out approximate inference for over-parameterized models, and ultimately show how advances in kernel methods can be translated into advances in approximate Bayesian inference.