# The Deep Weight Prior

arXiv: https://arxiv.org/abs/1810.06943 · HTML: https://mlsanity.com/paper/1810.06943/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2018.

## Built on by (most cited first, 11 of 11)

- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [A Bit More Bayesian: Domain-Invariant Learning with Uncertainty](https://mlsanity.com/paper/2105.04030/) (2021)
- [Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect](https://mlsanity.com/paper/2106.06596/) (2021)
- [Sparse Uncertainty Representation in Deep Learning with Inducing Weights](https://mlsanity.com/paper/2105.14594/) (2021)
- [Incorporating Unlabelled Data into Bayesian Neural Networks](https://mlsanity.com/paper/2304.01762/) (2023)
- [Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling](https://mlsanity.com/paper/2103.06198/) (2021)
- [Symmetric Wasserstein Autoencoders](https://mlsanity.com/paper/2106.13024/) (2021)
- [Minimum Description Length Control](https://mlsanity.com/paper/2207.08258/) (2022)
- [From Pointwise to Powerhouse: Initialising Neural Networks with Generative Models](https://mlsanity.com/paper/2310.16695/) (2023)
- [Reparameterization through Coverings and Topological Weight Priors](https://mlsanity.com/paper/2604.23804/) (2026)

## Abstract (arXiv)

Bayesian inference is known to provide a general framework for incorporating prior knowledge or specific properties into machine learning models via carefully choosing a prior distribution. In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior (DWP), that exploit generative models to encourage a specific structure of trained convolutional filters e.g., spatial correlations of weights. We define DWP in the form of an implicit distribution and propose a method for variational inference with such type of implicit priors. In experiments, we show that DWP improves the performance of Bayesian neural networks when training data are limited, and initialization of weights with samples from DWP accelerates training of conventional convolutional neural networks.