# Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models

arXiv: https://arxiv.org/abs/2103.04922 · HTML: https://mlsanity.com/paper/2103.04922/

## Key facts

- Cited by 252 papers in the map (data through 14 Oct 2026).
- Builds on 202 of its 276 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 30 of 202)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)

## Built on by (most cited first, 30 of 252)

- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [Human Motion Generation: A Survey](https://mlsanity.com/paper/2307.10894/) (2023)
- [Generative Diffusion Models on Graphs: Methods and Applications](https://mlsanity.com/paper/2302.02591/) (2023)
- [Synthcity: facilitating innovative use cases of synthetic data in different data modalities](https://mlsanity.com/paper/2301.07573/) (2023)
- [Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes](https://mlsanity.com/paper/2111.12701/) (2021)
- [BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis](https://mlsanity.com/paper/2203.13508/) (2022)
- [LiDAR Data Synthesis with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2309.09256/) (2023)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability](https://mlsanity.com/paper/2310.17467/) (2023)
- [Diffusion Model-Augmented Behavioral Cloning](https://mlsanity.com/paper/2302.13335/) (2023)
- [Synthetic Data for Face Recognition: Current State and Future Prospects](https://mlsanity.com/paper/2305.01021/) (2023)
- [SoK: Privacy-Preserving Data Synthesis](https://mlsanity.com/paper/2307.02106/) (2023)
- [Bilateral Denoising Diffusion Models](https://mlsanity.com/paper/2108.11514/) (2021)
- [A Study on the Evaluation of Generative Models](https://mlsanity.com/paper/2206.10935/) (2022)
- [Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Data](https://mlsanity.com/paper/2304.03722/) (2023)
- [Diagnosing and Fixing Manifold Overfitting in Deep Generative Models](https://mlsanity.com/paper/2204.07172/) (2022)
- [Modeling sRGB Camera Noise with Normalizing Flows](https://mlsanity.com/paper/2206.00812/) (2022)
- [Parallel Vertex Diffusion for Unified Visual Grounding](https://mlsanity.com/paper/2303.07216/) (2023)
- [Efficient-VDVAE: Less is more](https://mlsanity.com/paper/2203.13751/) (2022)
- [The Rise of Diffusion Models in Time-Series Forecasting](https://mlsanity.com/paper/2401.03006/) (2024)
- [Do Generated Data Always Help Contrastive Learning?](https://mlsanity.com/paper/2403.12448/) (2024)
- [scDiffusion: conditional generation of high-quality single-cell data using diffusion model](https://mlsanity.com/paper/2401.03968/) (2024)
- [Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark](https://mlsanity.com/paper/2310.16981/) (2023)
- [Transflower: probabilistic autoregressive dance generation with multimodal attention](https://mlsanity.com/paper/2106.13871/) (2021)
- [Phoenix: A Federated Generative Diffusion Model](https://mlsanity.com/paper/2306.04098/) (2023)

## Abstract (arXiv)

Deep generative models are a class of techniques that train deep neural networks to model the distribution of training samples. Research has fragmented into various interconnected approaches, each of which make trade-offs including run-time, diversity, and architectural restrictions. In particular, this compendium covers energy-based models, variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, in addition to numerous hybrid approaches. These techniques are compared and contrasted, explaining the premises behind each and how they are interrelated, while reviewing current state-of-the-art advances and implementations.