# Predictive Complexity Priors

arXiv: https://arxiv.org/abs/2006.10801 · HTML: https://mlsanity.com/paper/2006.10801/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 14 of 14)

- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Periodic Activation Functions Induce Stationarity](https://mlsanity.com/paper/2110.13572/) (2021)
- [Knowledge-Adaptation Priors](https://mlsanity.com/paper/2106.08769/) (2021)
- [Model Selection for Bayesian Autoencoders](https://mlsanity.com/paper/2106.06245/) (2021)
- [Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Prior](https://mlsanity.com/paper/2203.00479/) (2022)
- [Incorporating Unlabelled Data into Bayesian Neural Networks](https://mlsanity.com/paper/2304.01762/) (2023)
- [Bayesian Neural Networks with Domain Knowledge Priors](https://mlsanity.com/paper/2402.13410/) (2024)
- [Stochastic Deep Networks with Linear Competing Units for Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2208.01573/) (2022)
- [Regularized KL-Divergence for Well-Defined Function-Space Variational Inference in Bayesian neural networks](https://mlsanity.com/paper/2406.04317/) (2024)
- [Variational Bayesian Neural Networks via Resolution of Singularities](https://mlsanity.com/paper/2302.06035/) (2023)
- [Minimum Description Length Control](https://mlsanity.com/paper/2207.08258/) (2022)
- [On the meaning of uncertainty for ethical AI: philosophy and practice](https://mlsanity.com/paper/2309.05529/) (2023)
- [Joint Model and Data Sparsification via the Marginal Likelihood](https://mlsanity.com/paper/2605.29908/) (2026)

## Abstract (arXiv)

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model's predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model's predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.