# A note on the evaluation of generative models

arXiv: https://arxiv.org/abs/1511.01844 · HTML: https://mlsanity.com/paper/1511.01844/

## Key facts

- Cited by 292 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2015.

## Built on by (most cited first, 30 of 292)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2101.12072/) (2021)
- [Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.06503/) (2022)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [Challenges in Detoxifying Language Models](https://mlsanity.com/paper/2109.07445/) (2021)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [Aligning Language Models with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2302.08215/) (2023)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [StarVector: Generating Scalable Vector Graphics Code from Images and Text](https://mlsanity.com/paper/2312.11556/) (2023)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)
- [Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation](https://mlsanity.com/paper/2106.05527/) (2021)
- [KoLA: Carefully Benchmarking World Knowledge of Large Language Models](https://mlsanity.com/paper/2306.09296/) (2023)
- [Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models](https://mlsanity.com/paper/2206.07309/) (2022)
- [An Introduction to Neural Data Compression](https://mlsanity.com/paper/2202.06533/) (2022)
- [OmniSVG: A Unified Scalable Vector Graphics Generation Model](https://mlsanity.com/paper/2504.06263/) (2025)
- [Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes](https://mlsanity.com/paper/2111.12701/) (2021)
- [Predicting Video with VQVAE](https://mlsanity.com/paper/2103.01950/) (2021)
- [On Memorization in Probabilistic Deep Generative Models](https://mlsanity.com/paper/2106.03216/) (2021)
- [Hierarchical VAEs Know What They Don't Know](https://mlsanity.com/paper/2102.08248/) (2021)
- [Diffusion-Based Representation Learning](https://mlsanity.com/paper/2105.14257/) (2021)
- [Score-Based Generative Classifiers](https://mlsanity.com/paper/2110.00473/) (2021)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [On Aliased Resizing and Surprising Subtleties in GAN Evaluation](https://mlsanity.com/paper/2104.11222/) (2021)
- [Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning](https://mlsanity.com/paper/2303.16535/) (2023)

## Abstract (arXiv)

Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.