# Variational Lossy Autoencoder

arXiv: https://arxiv.org/abs/1611.02731 · HTML: https://mlsanity.com/paper/1611.02731/

## Key facts

- Cited by 176 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2016.

## Built on by (most cited first, 30 of 176)

- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis](https://mlsanity.com/paper/2108.08827/) (2021)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)
- [Mixed supervision for surface-defect detection: from weakly to fully supervised learning](https://mlsanity.com/paper/2104.06064/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)
- [D2C: Diffusion-Denoising Models for Few-shot Conditional Generation](https://mlsanity.com/paper/2106.06819/) (2021)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior](https://mlsanity.com/paper/2106.06406/) (2021)
- [GIVT: Generative Infinite-Vocabulary Transformers](https://mlsanity.com/paper/2312.02116/) (2023)
- [Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features](https://mlsanity.com/paper/2210.06756/) (2022)
- [Posterior Collapse and Latent Variable Non-identifiability](https://mlsanity.com/paper/2301.00537/) (2023)
- [SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer](https://mlsanity.com/paper/2412.10958/) (2024)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [Soft then Hard: Rethinking the Quantization in Neural Image Compression](https://mlsanity.com/paper/2104.05168/) (2021)
- [A-NeSI: A Scalable Approximate Method for Probabilistic Neurosymbolic Inference](https://mlsanity.com/paper/2212.12393/) (2022)
- [Stochastic Image-to-Video Synthesis using cINNs](https://mlsanity.com/paper/2105.04551/) (2021)
- [Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections](https://mlsanity.com/paper/2404.02954/) (2024)
- [iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis](https://mlsanity.com/paper/2107.02790/) (2021)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [Snowflake Point Deconvolution for Point Cloud Completion and Generation with Skip-Transformer](https://mlsanity.com/paper/2202.09367/) (2022)
- [Autoregressive Diffusion Transformer for Text-to-Speech Synthesis](https://mlsanity.com/paper/2406.05551/) (2024)
- [Autoregressive Conditional Neural Processes](https://mlsanity.com/paper/2303.14468/) (2023)
- [VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming](https://mlsanity.com/paper/2202.04178/) (2022)
- [Personalized Trajectory Prediction via Distribution Discrimination](https://mlsanity.com/paper/2107.14204/) (2021)
- [DisenHAN: Disentangled Heterogeneous Graph Attention Network for Recommendation](https://mlsanity.com/paper/2106.10879/) (2021)

## Abstract (arXiv)

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution p(z) and decoding distribution p(x|z), we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.