# Composing graphical models with neural networks for structured representations and fast inference

arXiv: https://arxiv.org/abs/1603.06277 · HTML: https://mlsanity.com/paper/1603.06277/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2016.

## Built on by (most cited first, 30 of 113)

- [Identifiability of deep generative models without auxiliary information](https://mlsanity.com/paper/2206.10044/) (2022)
- [Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA](https://mlsanity.com/paper/2106.09620/) (2021)
- [Posterior Collapse and Latent Variable Non-identifiability](https://mlsanity.com/paper/2301.00537/) (2023)
- [SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer](https://mlsanity.com/paper/2412.10958/) (2024)
- [Learning nonparametric latent causal graphs with unknown interventions](https://mlsanity.com/paper/2306.02899/) (2023)
- [Multi-Facet Clustering Variational Autoencoders](https://mlsanity.com/paper/2106.05241/) (2021)
- [A Deep Variational Approach to Clustering Survival Data](https://mlsanity.com/paper/2106.05763/) (2021)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Generative learning for nonlinear dynamics](https://mlsanity.com/paper/2311.04128/) (2023)
- [Deep Bayesian Unsupervised Lifelong Learning](https://mlsanity.com/paper/2106.07035/) (2021)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Amortized Variational Inference for Simple Hierarchical Models](https://mlsanity.com/paper/2111.03144/) (2021)
- [Deep Explicit Duration Switching Models for Time Series](https://mlsanity.com/paper/2110.13878/) (2021)
- [Emergent Communication through Metropolis-Hastings Naming Game with Deep Generative Models](https://mlsanity.com/paper/2205.12392/) (2022)
- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [Cauchy-Schwarz Regularized Autoencoder](https://mlsanity.com/paper/2101.02149/) (2021)
- [Concept Graph Neural Networks for Surgical Video Understanding](https://mlsanity.com/paper/2202.13402/) (2022)
- [Generative Causal Representation Learning for Out-of-Distribution Motion Forecasting](https://mlsanity.com/paper/2302.08635/) (2023)
- [ProbNeRF: Uncertainty-Aware Inference of 3D Shapes from 2D Images](https://mlsanity.com/paper/2210.17415/) (2022)
- [On Incorporating Inductive Biases into VAEs](https://mlsanity.com/paper/2106.13746/) (2021)
- [Neural Graphical Models](https://mlsanity.com/paper/2210.00453/) (2022)
- [Revisiting Structured Variational Autoencoders](https://mlsanity.com/paper/2305.16543/) (2023)
- [Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs](https://mlsanity.com/paper/2403.13748/) (2024)
- [Variational Inference for Continuous-Time Switching Dynamical Systems](https://mlsanity.com/paper/2109.14492/) (2021)
- [On the Identifiability of Switching Dynamical Systems](https://mlsanity.com/paper/2305.15925/) (2023)
- [Breiman's two cultures: You don't have to choose sides](https://mlsanity.com/paper/2104.12219/) (2021)
- [Unsupervised Learning of Neurosymbolic Encoders](https://mlsanity.com/paper/2107.13132/) (2021)
- [Toward a `Standard Model' of Machine Learning](https://mlsanity.com/paper/2108.07783/) (2021)
- [CompILE: Compositional Imitation Learning and Execution](https://mlsanity.com/paper/1812.01483/) (2018)
- [Lossless compression with state space models using bits back coding](https://mlsanity.com/paper/2103.10150/) (2021)

## Abstract (arXiv)

We propose a general modeling and inference framework that composes probabilistic graphical models with deep learning methods and combines their respective strengths. Our model family augments graphical structure in latent variables with neural network observation models. For inference, we extend variational autoencoders to use graphical model approximating distributions with recognition networks that output conjugate potentials. All components of these models are learned simultaneously with a single objective, giving a scalable algorithm that leverages stochastic variational inference, natural gradients, graphical model message passing, and the reparameterization trick. We illustrate this framework with several example models and an application to mouse behavioral phenotyping.