# Learning Invariances using the Marginal Likelihood

arXiv: https://arxiv.org/abs/1808.05563 · HTML: https://mlsanity.com/paper/1808.05563/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2018.

## Built on by (most cited first, 30 of 51)

- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning](https://mlsanity.com/paper/2104.04975/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [Residual Pathway Priors for Soft Equivariance Constraints](https://mlsanity.com/paper/2112.01388/) (2021)
- [Bayesian Model Selection, the Marginal Likelihood, and Generalization](https://mlsanity.com/paper/2202.11678/) (2022)
- [Time Series Contrastive Learning with Information-Aware Augmentations](https://mlsanity.com/paper/2303.11911/) (2023)
- [DEVIANT: Depth EquiVarIAnt NeTwork for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.10758/) (2022)
- [Learning Partial Equivariances from Data](https://mlsanity.com/paper/2110.10211/) (2021)
- [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://mlsanity.com/paper/2202.10638/) (2022)
- [Relaxing Equivariance Constraints with Non-stationary Continuous Filters](https://mlsanity.com/paper/2204.07178/) (2022)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix](https://mlsanity.com/paper/2101.07732/) (2021)
- [Grounding inductive biases in natural images:invariance stems from variations in data](https://mlsanity.com/paper/2106.05121/) (2021)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance](https://mlsanity.com/paper/2306.02866/) (2023)
- [Learning Layer-wise Equivariances Automatically using Gradients](https://mlsanity.com/paper/2310.06131/) (2023)
- [Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect](https://mlsanity.com/paper/2106.06596/) (2021)
- [Last Layer Marginal Likelihood for Invariance Learning](https://mlsanity.com/paper/2106.07512/) (2021)
- [Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learning](https://mlsanity.com/paper/2205.06342/) (2022)
- [Learning Invariant Weights in Neural Networks](https://mlsanity.com/paper/2202.12439/) (2022)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [Learning Augmentation Distributions using Transformed Risk Minimization](https://mlsanity.com/paper/2111.08190/) (2021)
- [A Complexity-Based Theory of Compositionality](https://mlsanity.com/paper/2410.14817/) (2024)
- [Modulated Neural ODEs](https://mlsanity.com/paper/2302.13262/) (2023)
- [Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels](https://mlsanity.com/paper/2306.03968/) (2023)
- [A Generative Model of Symmetry Transformations](https://mlsanity.com/paper/2403.01946/) (2024)
- [Regularising for invariance to data augmentation improves supervised learning](https://mlsanity.com/paper/2203.03304/) (2022)
- [Incorporating Unlabelled Data into Bayesian Neural Networks](https://mlsanity.com/paper/2304.01762/) (2023)
- [Neural Networks for Learning Counterfactual G-Invariances from Single Environments](https://mlsanity.com/paper/2104.10105/) (2021)

## Abstract (arXiv)

Generalising well in supervised learning tasks relies on correctly extrapolating the training data to a large region of the input space. One way to achieve this is to constrain the predictions to be invariant to transformations on the input that are known to be irrelevant (e.g. translation). Commonly, this is done through data augmentation, where the training set is enlarged by applying hand-crafted transformations to the inputs. We argue that invariances should instead be incorporated in the model structure, and learned using the marginal likelihood, which correctly rewards the reduced complexity of invariant models. We demonstrate this for Gaussian process models, due to the ease with which their marginal likelihood can be estimated. Our main contribution is a variational inference scheme for Gaussian processes containing invariances described by a sampling procedure. We learn the sampling procedure by back-propagating through it to maximise the marginal likelihood.