# Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders

arXiv: https://arxiv.org/abs/2109.12679 · HTML: https://mlsanity.com/paper/2109.12679/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 26 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [On the Fairness of Disentangled Representations](https://mlsanity.com/paper/1905.13662/) (2019)
- [Recent Advances in Autoencoder-Based Representation Learning](https://mlsanity.com/paper/1812.05069/) (2018)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](https://mlsanity.com/paper/1905.12506/) (2019)
- [On Disentangled Representations Learned From Correlated Data](https://mlsanity.com/paper/2006.07886/) (2020)
- [Variational Autoencoders Pursue PCA Directions (by Accident)](https://mlsanity.com/paper/1812.06775/) (2018)
- [The Usual Suspects? Reassessing Blame for VAE Posterior Collapse](https://mlsanity.com/paper/1912.10702/) (2019)
- [Revisiting Factorizing Aggregated Posterior in Learning Disentangled Representations](https://mlsanity.com/paper/2009.05739/) (2020)

## Built on by (most cited first, 7 of 7)

- [How do Variational Autoencoders Learn? Insights from Representational Similarity](https://mlsanity.com/paper/2205.08399/) (2022)
- [FONDUE: an algorithm to find the optimal dimensionality of the latent representations of variational autoencoders](https://mlsanity.com/paper/2209.12806/) (2022)
- [How good are variational autoencoders at transfer learning?](https://mlsanity.com/paper/2304.10767/) (2023)
- [Disentangled Latent Spaces for Reduced Order Models using Deterministic Autoencoders](https://mlsanity.com/paper/2502.14679/) (2025)
- [Disentanglement by means of action-induced representations](https://mlsanity.com/paper/2602.06741/) (2026)
- [Break The Spell Of Total Correlation In betaTCVAE](https://mlsanity.com/paper/2210.08794/) (2022)
- [Entropy-Based Characterisation of the Polarised Regime in Latent Variable Models](https://mlsanity.com/paper/2605.15965/) (2026)

## Abstract (arXiv)

The ability of Variational Autoencoders to learn disentangled representations has made them appealing for practical applications. However, their mean representations, which are generally used for downstream tasks, have recently been shown to be more correlated than their sampled counterpart, on which disentanglement is usually measured. In this paper, we refine this observation through the lens of selective posterior collapse, which states that only a subset of the learned representations, the active variables, is encoding useful information while the rest (the passive variables) is discarded. We first extend the existing definition to multiple data examples and show that active variables are equally disentangled in mean and sampled representations. Based on this extension and the pre-trained models from disentanglement lib, we then isolate the passive variables and show that they are responsible for the discrepancies between mean and sampled representations. Specifically, passive variables exhibit high correlation scores with other variables in mean representations while being fully uncorrelated in sampled ones. We thus conclude that despite what their higher correlation might suggest, mean representations are still good candidates for downstream tasks applications. However, it may be beneficial to remove their passive variables, especially when used with models sensitive to correlated features.