# PixelVAE: A Latent Variable Model for Natural Images

arXiv: https://arxiv.org/abs/1611.05013 · HTML: https://mlsanity.com/paper/1611.05013/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2016.

## Built on by (most cited first, 30 of 70)

- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis](https://mlsanity.com/paper/2108.08827/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Posterior Collapse and Latent Variable Non-identifiability](https://mlsanity.com/paper/2301.00537/) (2023)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [Autoregressive Conditional Neural Processes](https://mlsanity.com/paper/2303.14468/) (2023)
- [Rethinking Patch Dependence for Masked Autoencoders](https://mlsanity.com/paper/2401.14391/) (2024)
- [Deep Lossy Plus Residual Coding for Lossless and Near-lossless Image Compression](https://mlsanity.com/paper/2209.04847/) (2022)
- [Creativity and Machine Learning: A Survey](https://mlsanity.com/paper/2104.02726/) (2021)
- [CLUTR: Curriculum Learning via Unsupervised Task Representation Learning](https://mlsanity.com/paper/2210.10243/) (2022)
- [Learning Manifold Dimensions with Conditional Variational Autoencoders](https://mlsanity.com/paper/2302.11756/) (2023)
- [IRGen: Generative Modeling for Image Retrieval](https://mlsanity.com/paper/2303.10126/) (2023)
- [Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling](https://mlsanity.com/paper/2505.17384/) (2025)
- [Adapting Self-Supervised Representations as a Latent Space for Efficient Generation](https://mlsanity.com/paper/2510.14630/) (2025)
- [On Incorporating Inductive Biases into VAEs](https://mlsanity.com/paper/2106.13746/) (2021)
- [DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models](https://mlsanity.com/paper/2411.03250/) (2024)
- [Camera-Conditioned Stable Feature Generation for Isolated Camera Supervised Person Re-IDentification](https://mlsanity.com/paper/2203.15210/) (2022)
- [Unsupervised Object Representation Learning using Translation and Rotation Group Equivariant VAE](https://mlsanity.com/paper/2210.12918/) (2022)
- [Hierarchical VAE with a Diffusion-based VampPrior](https://mlsanity.com/paper/2412.01373/) (2024)
- [Variational Laplace Autoencoders](https://mlsanity.com/paper/2211.17267/) (2022)
- [DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards](https://mlsanity.com/paper/2304.10770/) (2023)
- [Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network](https://mlsanity.com/paper/2304.12770/) (2023)
- [Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders](https://mlsanity.com/paper/2306.05023/) (2023)
- [Boosting Flow-based Generative Super-Resolution Models via Learned Prior](https://mlsanity.com/paper/2403.10988/) (2024)
- [Improving VAE-based Representation Learning](https://mlsanity.com/paper/2205.14539/) (2022)
- [Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse](https://mlsanity.com/paper/2207.09535/) (2022)
- [Trading Information between Latents in Hierarchical Variational Autoencoders](https://mlsanity.com/paper/2302.04855/) (2023)
- [Learning Hierarchical Features with Joint Latent Space Energy-Based Prior](https://mlsanity.com/paper/2310.09604/) (2023)
- [Split Hierarchical Variational Compression](https://mlsanity.com/paper/2204.02071/) (2022)
- [Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks](https://mlsanity.com/paper/2208.09596/) (2022)

## Abstract (arXiv)

Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64x64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.