# Overlooked Implications of the Reconstruction Loss for VAE Disentanglement

arXiv: https://arxiv.org/abs/2202.13341 · HTML: https://mlsanity.com/paper/2202.13341/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 27 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [On the Fairness of Disentangled Representations](https://mlsanity.com/paper/1905.13662/) (2019)
- [InfoVAE: Information Maximizing Variational Autoencoders](https://mlsanity.com/paper/1706.02262/) (2017)
- [On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset](https://mlsanity.com/paper/1906.03292/) (2019)
- [Variational Autoencoders Pursue PCA Directions (by Accident)](https://mlsanity.com/paper/1812.06775/) (2018)
- [Measuring Disentanglement: A Review of Metrics](https://mlsanity.com/paper/2012.09276/) (2020)
- [On the Transfer of Disentangled Representations in Realistic Settings](https://mlsanity.com/paper/2010.14407/) (2020)
- [Demystifying Inductive Biases for β-VAE Based Architectures](https://mlsanity.com/paper/2102.06822/) (2021)

## Built on by (most cited first, 7 of 7)

- [Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks](https://mlsanity.com/paper/2305.01713/) (2023)
- [Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders](https://mlsanity.com/paper/2311.08579/) (2023)
- [Disentanglement Learning via Topology](https://mlsanity.com/paper/2308.12696/) (2023)
- [Disentangling Disentangled Representations: Towards Improved Latent Units via Diffusion Models](https://mlsanity.com/paper/2410.23820/) (2024)
- [Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks](https://mlsanity.com/paper/2503.06181/) (2025)
- [Unveiling and Steering Connectome Organization with Interpretable Latent Variables](https://mlsanity.com/paper/2505.13011/) (2025)
- [Exploring Image Generation via Mutually Exclusive Probability Spaces and Local Correlation Hypothesis](https://mlsanity.com/paper/2506.21731/) (2025)

## Abstract (arXiv)

Learning disentangled representations with variational autoencoders (VAEs) is often attributed to the regularisation component of the loss. In this work, we highlight the interaction between data and the reconstruction term of the loss as the main contributor to disentanglement in VAEs. We show that standard benchmark datasets have unintended correlations between their subjective ground-truth factors and perceived axes in the data according to typical VAE reconstruction losses. Our work exploits this relationship to provide a theory for what constitutes an adversarial dataset under a given reconstruction loss. We verify this by constructing an example dataset that prevents disentanglement in state-of-the-art frameworks while maintaining human-intuitive ground-truth factors. Finally, we re-enable disentanglement by designing an example reconstruction loss that is once again able to perceive the ground-truth factors. Our findings demonstrate the subjective nature of disentanglement and the importance of considering the interaction between the ground-truth factors, data and notably, the reconstruction loss, which is under-recognised in the literature.