# Trading Information between Latents in Hierarchical Variational Autoencoders

arXiv: https://arxiv.org/abs/2302.04855 · HTML: https://mlsanity.com/paper/2302.04855/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications](https://mlsanity.com/paper/1802.03903/) (2018)
- [Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering](https://mlsanity.com/paper/1611.05148/) (2016)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling](https://mlsanity.com/paper/1902.02102/) (2019)
- [Taming VAEs](https://mlsanity.com/paper/1810.00597/) (2018)
- [Improving Inference for Neural Image Compression](https://mlsanity.com/paper/2006.04240/) (2020)
- [PixelVAE: A Latent Variable Model for Natural Images](https://mlsanity.com/paper/1611.05013/) (2016)
- [Universally Quantized Neural Compression](https://mlsanity.com/paper/2006.09952/) (2020)
- [Inversion using a new low-dimensional representation of complex binary geological media based on a deep neural network](https://mlsanity.com/paper/1710.09196/) (2017)

## Built on by (most cited first, 6 of 6)

- [Tree Variational Autoencoders](https://mlsanity.com/paper/2306.08984/) (2023)
- [Weakly Augmented Variational Autoencoder in Time Series Anomaly Detection](https://mlsanity.com/paper/2401.03341/) (2024)
- [ParamReL: Learning Parameter Space Representation via Progressively Encoding Bayesian Flow Networks](https://mlsanity.com/paper/2405.15268/) (2024)
- [A Note on Generalization in Variational Autoencoders: How Effective Is Synthetic Data & Overparameterization?](https://mlsanity.com/paper/2310.19653/) (2023)
- [Your Image is Secretly the Last Frame of a Pseudo Video](https://mlsanity.com/paper/2410.20158/) (2024)
- [H"older++: Improving the Quality-Coherence Trade-off in Multimodal VAEs](https://mlsanity.com/paper/2606.13381/) (2026)

## Abstract (arXiv)

Variational Autoencoders (VAEs) were originally motivated (Kingma & Welling, 2014) as probabilistic generative models in which one performs approximate Bayesian inference. The proposal of β-VAEs (Higgins et al., 2017) breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing an objective function that allows practitioners to trade off between the information content ("bit rate") of the latent representation and the distortion of reconstructed data (Alemi et al., 2018). In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.e., VAEs with more than one layer of latent variables. We identify a general class of inference models for which one can split the rate into contributions from each layer, which can then be tuned independently. We derive theoretical bounds on the performance of downstream tasks as functions of the individual layers' rates and verify our theoretical findings in large-scale experiments. Our results provide guidance for practitioners on which region in rate-space to target for a given application.