# Penalising model component complexity: A principled, practical approach to constructing priors

arXiv: https://arxiv.org/abs/1403.4630 · HTML: https://mlsanity.com/paper/1403.4630/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2014.

## Built on by (most cited first, 13 of 13)

- [LMPriors: Pre-Trained Language Models as Task-Specific Priors](https://mlsanity.com/paper/2210.12530/) (2022)
- [Knowledge-Adaptation Priors](https://mlsanity.com/paper/2106.08769/) (2021)
- [Fast Mixing of Data Augmentation Algorithms: Bayesian Probit, Logit, and Lasso Regression](https://mlsanity.com/paper/2412.07999/) (2024)
- [BAM: Bayes with Adaptive Memory](https://mlsanity.com/paper/2202.02405/) (2022)
- [Hierarchical Embedded Bayesian Additive Regression Trees](https://mlsanity.com/paper/2204.07207/) (2022)
- [Mixing times of data-augmentation Gibbs samplers for high-dimensional probit regression](https://mlsanity.com/paper/2505.14343/) (2025)
- [Expert-elicitation method for non-parametric joint priors using normalizing flows](https://mlsanity.com/paper/2411.15826/) (2024)
- [Wasserstein Contraction of Coordinate Ascent Variational Inference](https://mlsanity.com/paper/2605.30253/) (2026)
- [Fast-Mixing Markov Chains without Gradients](https://mlsanity.com/paper/2606.27564/) (2026)
- [A Practitioner's Guide to Bayesian Inference in Pharmacometrics using Pumas](https://mlsanity.com/paper/2304.04752/) (2023)
- [Semi-Parametric Inference for Doubly Stochastic Spatial Point Processes: An Approximate Penalized Poisson Likelihood Approach](https://mlsanity.com/paper/2306.06756/) (2023)
- [Modeling High-Resolution Spatio-Temporal Wind with Deep Echo State Networks and Stochastic Partial Differential Equations](https://mlsanity.com/paper/2412.07265/) (2024)
- [Bayesian Semi-Parametric Spatial Dispersed Count Model for Precipitation Analysis](https://mlsanity.com/paper/2503.19117/) (2025)

## Abstract (arXiv)

In this paper, we introduce a new concept for constructing prior distributions. We exploit the natural nested structure inherent to many model components, which defines the model component to be a flexible extension of a base model. Proper priors are defined to penalise the complexity induced by deviating from the simpler base model and are formulated after the input of a user-defined scaling parameter for that model component, both in the univariate and the multivariate case. These priors are invariant to reparameterisations, have a natural connection to Jeffreys' priors, are designed to support Occam's razor and seem to have excellent robustness properties, all which are highly desirable and allow us to use this approach to define default prior distributions. Through examples and theoretical results, we demonstrate the appropriateness of this approach and how it can be applied in various situations.