# Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling

arXiv: https://arxiv.org/abs/2110.06021 · HTML: https://mlsanity.com/paper/2110.06021/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Flow-based generative models for Markov chain Monte Carlo in lattice field theory](https://mlsanity.com/paper/1904.12072/) (2019)
- [Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows](https://mlsanity.com/paper/2002.06103/) (2020)
- [Equivariant flow-based sampling for lattice gauge theory](https://mlsanity.com/paper/2003.06413/) (2020)
- [Implicit Reparameterization Gradients](https://mlsanity.com/paper/1805.08498/) (2018)
- [An Introduction to Probabilistic Programming](https://mlsanity.com/paper/1809.10756/) (2018)
- [Equivariant Hamiltonian Flows](https://mlsanity.com/paper/1909.13739/) (2019)
- [Recurrent Highway Networks](https://mlsanity.com/paper/1607.03474/) (2016)
- [Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows](https://mlsanity.com/paper/2002.10516/) (2020)
- [Graphical Normalizing Flows](https://mlsanity.com/paper/2006.02548/) (2020)
- [Introduction to Normalizing Flows for Lattice Field Theory](https://mlsanity.com/paper/2101.08176/) (2021)
- [Edward: A library for probabilistic modeling, inference, and criticism](https://mlsanity.com/paper/1610.09787/) (2016)
- [Deep Probabilistic Programming](https://mlsanity.com/paper/1701.03757/) (2017)
- [Automatic structured variational inference](https://mlsanity.com/paper/2002.00643/) (2020)
- [Automatic Reparameterisation of Probabilistic Programs](https://mlsanity.com/paper/1906.03028/) (2019)
- [Neural Canonical Transformation with Symplectic Flows](https://mlsanity.com/paper/1910.00024/) (2019)
- [Automatic variational inference with cascading flows](https://mlsanity.com/paper/2102.04801/) (2021)
- [Joint Distributions for TensorFlow Probability](https://mlsanity.com/paper/2001.11819/) (2020)

## Built on by (most cited first, 3 of 3)

- [Structured Neural Networks for Density Estimation and Causal Inference](https://mlsanity.com/paper/2311.02221/) (2023)
- [Model-Informed Flows for Bayesian Inference](https://mlsanity.com/paper/2505.24243/) (2025)
- [Causal Posterior Estimation](https://mlsanity.com/paper/2505.21468/) (2025)

## Abstract (arXiv)

Normalizing flows have shown great success as general-purpose density estimators. However, many real world applications require the use of domain-specific knowledge, which normalizing flows cannot readily incorporate. We propose embedded-model flows (EMF), which alternate general-purpose transformations with structured layers that embed domain-specific inductive biases. These layers are automatically constructed by converting user-specified differentiable probabilistic models into equivalent bijective transformations. We also introduce gated structured layers, which allow bypassing the parts of the models that fail to capture the statistics of the data. We demonstrate that EMFs can be used to induce desirable properties such as multimodality, hierarchical coupling and continuity. Furthermore, we show that EMFs enable a high performance form of variational inference where the structure of the prior model is embedded in the variational architecture. In our experiments, we show that this approach outperforms state-of-the-art methods in common structured inference problems.