# Marginalizable Density Models

arXiv: https://arxiv.org/abs/2106.04741 · HTML: https://mlsanity.com/paper/2106.04741/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 78 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [Learning Equivalence Classes of Bayesian Networks Structures](https://mlsanity.com/paper/1302.3566/) (2013)
- [Approximate Kernel-based Conditional Independence Tests for Fast Non-Parametric Causal Discovery](https://mlsanity.com/paper/1702.03877/) (2017)
- [Deep Multi-task Representation Learning: A Tensor Factorisation Approach](https://mlsanity.com/paper/1605.06391/) (2016)
- [Block Neural Autoregressive Flow](https://mlsanity.com/paper/1904.04676/) (2019)
- [Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives](https://mlsanity.com/paper/1708.09165/) (2017)
- [Causal Discovery Toolbox: Uncover causal relationships in Python](https://mlsanity.com/paper/1903.02278/) (2019)
- [Transformation Autoregressive Networks](https://mlsanity.com/paper/1801.09819/) (2018)
- [Neural Likelihoods via Cumulative Distribution Functions](https://mlsanity.com/paper/1811.00974/) (2018)
- [Global Optimality in Tensor Factorization, Deep Learning, and Beyond](https://mlsanity.com/paper/1506.07540/) (2015)
- [Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?](https://mlsanity.com/paper/2007.01038/) (2020)
- [Probit transformation for nonparametric kernel estimation of the copula density](https://mlsanity.com/paper/1404.4414/) (2014)
- [Learning Generative Models using Denoising Density Estimators](https://mlsanity.com/paper/2001.02728/) (2020)
- [Anomaly Detection in the Presence of Missing Values](https://mlsanity.com/paper/1809.01605/) (2018)
- [Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods](https://mlsanity.com/paper/1506.08473/) (2015)
- [A review of some recent advances in causal inference](https://mlsanity.com/paper/1506.07669/) (2015)

## Built on by (most cited first, 4 of 4)

- [Online Neural Path Guiding with Normalized Anisotropic Spherical Gaussians](https://mlsanity.com/paper/2303.08064/) (2023)
- [Local Flow Matching Generative Models](https://mlsanity.com/paper/2410.02548/) (2024)
- [AdaCat: Adaptive Categorical Discretization for Autoregressive Models](https://mlsanity.com/paper/2208.02246/) (2022)
- [TERM Model: Tensor Ring Mixture Model for Density Estimation](https://mlsanity.com/paper/2312.08075/) (2023)

## Abstract (arXiv)

Probability density models based on deep networks have achieved remarkable success in modeling complex high-dimensional datasets. However, unlike kernel density estimators, modern neural models do not yield marginals or conditionals in closed form, as these quantities require the evaluation of seldom tractable integrals. In this work, we present the Marginalizable Density Model Approximator (MDMA), a novel deep network architecture which provides closed form expressions for the probabilities, marginals and conditionals of any subset of the variables. The MDMA learns deep scalar representations for each individual variable and combines them via learned hierarchical tensor decompositions into a tractable yet expressive CDF, from which marginals and conditional densities are easily obtained. We illustrate the advantage of exact marginalizability in several tasks that are out of reach of previous deep network-based density estimation models, such as estimating mutual information between arbitrary subsets of variables, inferring causality by testing for conditional independence, and inference with missing data without the need for data imputation, outperforming state-of-the-art models on these tasks. The model also allows for parallelized sampling with only a logarithmic dependence of the time complexity on the number of variables.