# Encoding prior knowledge in the structure of the likelihood

arXiv: https://arxiv.org/abs/1812.04403 · HTML: https://mlsanity.com/paper/1812.04403/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2018.

## Built on by (most cited first, 3 of 3)

- [Probabilistic Autoencoder using Fisher Information](https://mlsanity.com/paper/2110.14947/) (2021)
- [Information Field Theory and Artificial Intelligence](https://mlsanity.com/paper/2112.10133/) (2021)
- [Causal, Bayesian, & Non-parametric Modeling of the SARS-CoV-2 Viral Load Distribution vs. Patient's Age](https://mlsanity.com/paper/2105.13483/) (2021)

## Abstract (arXiv)

The inference of deep hierarchical models is problematic due to strong dependencies between the hierarchies. We investigate a specific transformation of the model parameters based on the multivariate distributional transform. This transformation is a special form of the reparametrization trick, flattens the hierarchy and leads to a standard Gaussian prior on all resulting parameters. The transformation also transfers all the prior information into the structure of the likelihood, hereby decoupling the transformed parameters a priori from each other. A variational Gaussian approximation in this standardized space will be excellent in situations of relatively uninformative data. Additionally, the curvature of the log-posterior is well-conditioned in directions that are weakly constrained by the data, allowing for fast inference in such a scenario. In an example we perform the transformation explicitly for Gaussian process regression with a priori unknown correlation structure. Deep models are inferred rapidly in highly and slowly in poorly informed situations. The flat model show exactly the opposite performance pattern. A synthesis of both, the deep and the flat perspective, provides their combined advantages and overcomes the individual limitations, leading to a faster inference.