# Generalization Gap in Amortized Inference

arXiv: https://arxiv.org/abs/2205.11640 · HTML: https://mlsanity.com/paper/2205.11640/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [InfoVAE: Information Maximizing Variational Autoencoders](https://mlsanity.com/paper/1706.02262/) (2017)
- [Practical Lossless Compression with Latent Variables using Bits Back Coding](https://mlsanity.com/paper/1901.04866/) (2019)
- [Bias and Generalization in Deep Generative Models: An Empirical Study](https://mlsanity.com/paper/1811.03259/) (2018)
- [Improving Inference for Neural Image Compression](https://mlsanity.com/paper/2006.04240/) (2020)
- [Auxiliary Deep Generative Models](https://mlsanity.com/paper/1602.05473/) (2016)
- [Frequentist Consistency of Variational Bayes](https://mlsanity.com/paper/1705.03439/) (2017)
- [Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables](https://mlsanity.com/paper/1905.06845/) (2019)
- [HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models](https://mlsanity.com/paper/1912.09953/) (2019)
- [On the Out-of-distribution Generalization of Probabilistic Image Modelling](https://mlsanity.com/paper/2109.02639/) (2021)
- [Stochastic Backpropagation and Approximate Inference in Deep Generative Models](https://mlsanity.com/paper/1401.4082/) (2014)
- [Amortized Inference Regularization](https://mlsanity.com/paper/1805.08913/) (2018)
- [Autoencoding Variational Autoencoder](https://mlsanity.com/paper/2012.03715/) (2020)
- [Spread Divergence](https://mlsanity.com/paper/1811.08968/) (2018)
- [Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding](https://mlsanity.com/paper/2102.11086/) (2021)
- [Parallel Neural Local Lossless Compression](https://mlsanity.com/paper/2201.05213/) (2022)

## Built on by (most cited first, 10 of 10)

- [Computationally-Efficient Neural Image Compression with Shallow Decoders](https://mlsanity.com/paper/2304.06244/) (2023)
- [Amortized Variational Inference: A Systematic Review](https://mlsanity.com/paper/2209.10888/) (2022)
- [Laplacian Autoencoders for Learning Stochastic Representations](https://mlsanity.com/paper/2206.15078/) (2022)
- [Moment Matching Denoising Gibbs Sampling](https://mlsanity.com/paper/2305.11650/) (2023)
- [Improving VAE-based Representation Learning](https://mlsanity.com/paper/2205.14539/) (2022)
- [Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation](https://mlsanity.com/paper/2603.28744/) (2026)
- [Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling](https://mlsanity.com/paper/2308.09078/) (2023)
- [Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Families](https://mlsanity.com/paper/2403.03069/) (2024)
- [Amortized Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/2409.12301/) (2024)
- [A Note on Generalization in Variational Autoencoders: How Effective Is Synthetic Data & Overparameterization?](https://mlsanity.com/paper/2310.19653/) (2023)

## Abstract (arXiv)

The ability of likelihood-based probabilistic models to generalize to unseen data is central to many machine learning applications such as lossless compression. In this work, we study the generalization of a popular class of probabilistic model - the Variational Auto-Encoder (VAE). We discuss the two generalization gaps that affect VAEs and show that overfitting is usually dominated by amortized inference. Based on this observation, we propose a new training objective that improves the generalization of amortized inference. We demonstrate how our method can improve performance in the context of image modeling and lossless compression.