# Improving VAE-based Representation Learning

arXiv: https://arxiv.org/abs/2205.14539 · HTML: https://mlsanity.com/paper/2205.14539/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 73 references that are in the map.
- First posted on arXiv 28 May 2022.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [Why Normalizing Flows Fail to Detect Out-of-Distribution Data](https://mlsanity.com/paper/2006.08545/) (2020)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Unsupervised Learning by Predicting Noise](https://mlsanity.com/paper/1704.05310/) (2017)

## Built on by (most cited first, 6 of 6)

- [CLUTR: Curriculum Learning via Unsupervised Task Representation Learning](https://mlsanity.com/paper/2210.10243/) (2022)
- [A Probabilistic Model Behind Self-Supervised Learning](https://mlsanity.com/paper/2402.01399/) (2024)
- [Structural Adversarial Objectives for Self-Supervised Representation Learning](https://mlsanity.com/paper/2310.00357/) (2023)
- [Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior](https://mlsanity.com/paper/2410.10180/) (2024)
- [Learning Sparse Visual Representations via Spatial-Semantic Factorization](https://mlsanity.com/paper/2602.01905/) (2026)
- [A Saliency-based Clustering Framework for Identifying Aberrant Predictions](https://mlsanity.com/paper/2311.06454/) (2023)

## Abstract (arXiv)

Latent variable models like the Variational Auto-Encoder (VAE) are commonly used to learn representations of images. However, for downstream tasks like semantic classification, the representations learned by VAE are less competitive than other non-latent variable models. This has led to some speculations that latent variable models may be fundamentally unsuitable for representation learning. In this work, we study what properties are required for good representations and how different VAE structure choices could affect the learned properties. We show that by using a decoder that prefers to learn local features, the remaining global features can be well captured by the latent, which significantly improves performance of a downstream classification task. We further apply the proposed model to semi-supervised learning tasks and demonstrate improvements in data efficiency.