# Variational Gibbs Inference for Statistical Model Estimation from Incomplete Data

arXiv: https://arxiv.org/abs/2111.13180 · HTML: https://mlsanity.com/paper/2111.13180/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 92 references that are in the map.
- First posted on arXiv 25 Nov 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [GAIN: Missing Data Imputation using Generative Adversarial Nets](https://mlsanity.com/paper/1806.02920/) (2018)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [BRITS: Bidirectional Recurrent Imputation for Time Series](https://mlsanity.com/paper/1805.10572/) (2018)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Missing Data Imputation using Optimal Transport](https://mlsanity.com/paper/2002.03860/) (2020)
- [Oops I Took A Gradient: Scalable Sampling for Discrete Distributions](https://mlsanity.com/paper/2102.04509/) (2021)
- [EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE](https://mlsanity.com/paper/1809.11142/) (2018)
- [MisGAN: Learning from Incomplete Data with Generative Adversarial Networks](https://mlsanity.com/paper/1902.09599/) (2019)
- [not-MIWAE: Deep Generative Modelling with Missing not at Random Data](https://mlsanity.com/paper/2006.12871/) (2020)
- [Flow Models for Arbitrary Conditional Likelihoods](https://mlsanity.com/paper/1909.06319/) (2019)
- [VAEs in the Presence of Missing Data](https://mlsanity.com/paper/2006.05301/) (2020)
- [Identifiable Generative Models for Missing Not at Random Data Imputation](https://mlsanity.com/paper/2110.14708/) (2021)
- [Full Law Identification In Graphical Models Of Missing Data: Completeness Results](https://mlsanity.com/paper/2004.04872/) (2020)
- [Identification In Missing Data Models Represented By Directed Acyclic Graphs](https://mlsanity.com/paper/1907.00241/) (2019)
- [Proximity Variational Inference](https://mlsanity.com/paper/1705.08931/) (2017)
- [Autoencoders and Probabilistic Inference with Missing Data: An Exact Solution for The Factor Analysis Case](https://mlsanity.com/paper/1801.03851/) (2018)
- [Adapting The Gibbs Sampler](https://mlsanity.com/paper/1801.09299/) (2018)

## Built on by (most cited first, 4 of 4)

- [Delta-AI: Local objectives for amortized inference in sparse graphical models](https://mlsanity.com/paper/2310.02423/) (2023)
- [Variational Mixture of HyperGenerators for Learning Distributions Over Functions](https://mlsanity.com/paper/2302.06223/) (2023)
- [Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling](https://mlsanity.com/paper/2308.09078/) (2023)
- [Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Families](https://mlsanity.com/paper/2403.03069/) (2024)

## Abstract (arXiv)

Statistical models are central to machine learning with broad applicability across a range of downstream tasks. The models are controlled by free parameters that are typically estimated from data by maximum-likelihood estimation or approximations thereof. However, when faced with real-world data sets many of the models run into a critical issue: they are formulated in terms of fully-observed data, whereas in practice the data sets are plagued with missing data. The theory of statistical model estimation from incomplete data is conceptually similar to the estimation of latent-variable models, where powerful tools such as variational inference (VI) exist. However, in contrast to standard latent-variable models, parameter estimation with incomplete data often requires estimating exponentially-many conditional distributions of the missing variables, hence making standard VI methods intractable. We address this gap by introducing variational Gibbs inference (VGI), a new general-purpose method to estimate the parameters of statistical models from incomplete data. We validate VGI on a set of synthetic and real-world estimation tasks, estimating important machine learning models such as variational autoencoders and normalising flows from incomplete data. The proposed method, whilst general-purpose, achieves competitive or better performance than existing model-specific estimation methods.