# Understanding Variational Autoencoders with Intrinsic Dimension and Information Imbalance

arXiv: https://arxiv.org/abs/2411.01978 · HTML: https://mlsanity.com/paper/2411.01978/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning](https://mlsanity.com/paper/1803.04765/) (2018)
- [Latent Space Oddity: on the Curvature of Deep Generative Models](https://mlsanity.com/paper/1710.11379/) (2017)
- [InfoVAE: Information Maximizing Variational Autoencoders](https://mlsanity.com/paper/1706.02262/) (2017)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks](https://mlsanity.com/paper/2111.13171/) (2021)
- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [Metrics for Deep Generative Models](https://mlsanity.com/paper/1711.01204/) (2017)
- [Distributional Semantics and Linguistic Theory](https://mlsanity.com/paper/1905.01896/) (2019)
- [Learning Flat Latent Manifolds with VAEs](https://mlsanity.com/paper/2002.04881/) (2020)
- [Understanding Autoencoders with Information Theoretic Concepts](https://mlsanity.com/paper/1804.00057/) (2018)
- [Ranking the information content of distance measures](https://mlsanity.com/paper/2104.15079/) (2021)
- [DADApy: Distance-based Analysis of DAta-manifolds in Python](https://mlsanity.com/paper/2205.03373/) (2022)
- [On the Information Plane of Autoencoders](https://mlsanity.com/paper/2005.07783/) (2020)
- [Exploring the Latent Space of Autoencoders with Interventional Assays](https://mlsanity.com/paper/2106.16091/) (2021)
- [On Linear Interpolation in the Latent Space of Deep Generative Models](https://mlsanity.com/paper/2105.03663/) (2021)

## Built on by (most cited first, 1 of 1)

- [PixSDS: Why Latent SDS Makes Noisy Pixels](https://mlsanity.com/paper/2608.12997/) (2026)

## Abstract (arXiv)

This work presents an analysis of the hidden representations of Variational Autoencoders (VAEs) using the Intrinsic Dimension (ID) and the Information Imbalance (II). We show that VAEs undergo a transition in behaviour once the bottleneck size is larger than the ID of the data, manifesting in a double hunchback ID profile and a qualitative shift in information processing as captured by the II. Our results also highlight two distinct training phases for architectures with sufficiently large bottleneck sizes, consisting of a rapid fit and a slower generalisation, as assessed by a differentiated behaviour of ID, II, and KL loss. These insights demonstrate that II and ID could be valuable tools for aiding architecture search, for diagnosing underfitting in VAEs, and, more broadly, they contribute to advancing a unified understanding of deep generative models through geometric analysis.