# Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case

arXiv: https://arxiv.org/abs/2206.08309 · HTML: https://mlsanity.com/paper/2206.08309/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 78 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 30 of 43)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Variational Lossy Autoencoder](https://mlsanity.com/paper/1611.02731/) (2016)
- [Latent Space Oddity: on the Curvature of Deep Generative Models](https://mlsanity.com/paper/1710.11379/) (2017)
- [Effectively Unbiased FID and Inception Score and where to find them](https://mlsanity.com/paper/1911.07023/) (2019)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [InfoVAE: Information Maximizing Variational Autoencoders](https://mlsanity.com/paper/1706.02262/) (2017)
- [Continuous Hierarchical Representations with Poincar'e Variational Auto-Encoders](https://mlsanity.com/paper/1901.06033/) (2019)

## Built on by (most cited first, 26 of 26)

- [Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve](https://mlsanity.com/paper/2212.03905/) (2022)
- [Lifting Architectural Constraints of Injective Flows](https://mlsanity.com/paper/2306.01843/) (2023)
- [Evaluation of pseudo-healthy image reconstruction for anomaly detection with deep generative models: Application to brain FDG PET](https://mlsanity.com/paper/2401.16363/) (2024)
- [An efficient probabilistic hardware architecture for diffusion-like models](https://mlsanity.com/paper/2510.23972/) (2025)
- [Boosting Flow-based Generative Super-Resolution Models via Learned Prior](https://mlsanity.com/paper/2403.10988/) (2024)
- [Can3Tok: Canonical 3D Tokenization and Latent Modeling of Scene-Level 3D Gaussians](https://mlsanity.com/paper/2508.01464/) (2025)
- [Risk-Controlling Model Selection via Guided Bayesian Optimization](https://mlsanity.com/paper/2312.01692/) (2023)
- [One-Line-of-Code Data Mollification Improves Optimization of Likelihood-based Generative Models](https://mlsanity.com/paper/2305.18900/) (2023)
- [α-TCVAE: On the relationship between Disentanglement and Diversity](https://mlsanity.com/paper/2411.00588/) (2024)
- [LangVAE and LangSpace: Building and Probing for Language Model VAEs](https://mlsanity.com/paper/2505.00004/) (2025)
- [Modeling Complex Disease Trajectories using Deep Generative Models with Semi-Supervised Latent Processes](https://mlsanity.com/paper/2311.08149/) (2023)
- [Analyzing Generative Models by Manifold Entropic Metrics](https://mlsanity.com/paper/2410.19426/) (2024)
- [Geodesic Calculus on Implicitly Defined Latent Manifolds](https://mlsanity.com/paper/2510.09468/) (2025)
- [TC-VAE: Uncovering Out-of-Distribution Data Generative Factors](https://mlsanity.com/paper/2304.04103/) (2023)
- [Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis](https://mlsanity.com/paper/2305.11832/) (2023)
- [Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventions](https://mlsanity.com/paper/2505.16311/) (2025)
- [GMM-IL: Image Classification using Incrementally Learnt, Independent Probabilistic Models for Small Sample Sizes](https://mlsanity.com/paper/2212.00572/) (2022)
- [Simplex Autoencoders](https://mlsanity.com/paper/2301.06489/) (2023)
- [Variational Inference for Longitudinal Data Using Normalizing Flows](https://mlsanity.com/paper/2303.14220/) (2023)
- [Sampling From Autoencoders' Latent Space via Quantization And Probability Mass Function Concepts](https://mlsanity.com/paper/2308.10704/) (2023)
- [Semi-Supervised Generative Models for Disease Trajectories: A Case Study on Systemic Sclerosis](https://mlsanity.com/paper/2407.11427/) (2024)
- [An Empirical Study on Improving SimCLR's Nonlinear Projection Head using Pretrained Autoencoder Embeddings](https://mlsanity.com/paper/2408.14514/) (2024)
- [Bidirectional Variational Autoencoders](https://mlsanity.com/paper/2505.16074/) (2025)
- [Learning a Latent Pulse Shape Interface for Photoinjector Laser Systems](https://mlsanity.com/paper/2602.17263/) (2026)
- [How Neural Losses Shape VAE Latents](https://mlsanity.com/paper/2606.00635/) (2026)
- [A Multi-stage Constrained Optimization Framework for Data-driven Problems](https://mlsanity.com/paper/2607.23480/) (2026)

## Abstract (arXiv)

In recent years, deep generative models have attracted increasing interest due to their capacity to model complex distributions. Among those models, variational autoencoders have gained popularity as they have proven both to be computationally efficient and yield impressive results in multiple fields. Following this breakthrough, extensive research has been done in order to improve the original publication, resulting in a variety of different VAE models in response to different tasks. In this paper we present Pythae, a versatile open-source Python library providing both a unified implementation and a dedicated framework allowing straightforward, reproducible and reliable use of generative autoencoder models. We then propose to use this library to perform a case study benchmark where we present and compare 19 generative autoencoder models representative of some of the main improvements on downstream tasks such as image reconstruction, generation, classification, clustering and interpolation. The open-source library can be found at https://github.com/clementchadebec/benchmark_VAE.