# A Unifying Information-theoretic Perspective on Evaluating Generative Models

arXiv: https://arxiv.org/abs/2412.14340 · HTML: https://mlsanity.com/paper/2412.14340/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 18 of 18)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models](https://mlsanity.com/paper/1904.01121/) (2019)
- [Automatic Evaluation and Moderation of Open-domain Dialogue Systems](https://mlsanity.com/paper/2111.02110/) (2021)
- [Assessing Dialogue Systems with Distribution Distances](https://mlsanity.com/paper/2105.02573/) (2021)
- [Direct Estimation of Information Divergence Using Nearest Neighbor Ratios](https://mlsanity.com/paper/1702.05222/) (2017)
- [Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models](https://mlsanity.com/paper/2309.01590/) (2023)
- [Understanding Deep Generative Models with Generalized Empirical Likelihoods](https://mlsanity.com/paper/2306.09780/) (2023)

## Abstract (arXiv)

Considering the difficulty of interpreting generative model output, there is significant current research focused on determining meaningful evaluation metrics. Several recent approaches utilize "precision" and "recall," borrowed from the classification domain, to individually quantify the output fidelity (realism) and output diversity (representation of the real data variation), respectively. With the increase in metric proposals, there is a need for a unifying perspective, allowing for easier comparison and clearer explanation of their benefits and drawbacks. To this end, we unify a class of kth-nearest-neighbors (kNN)-based metrics under an information-theoretic lens using approaches from kNN density estimation. Additionally, we propose a tri-dimensional metric composed of Precision Cross-Entropy (PCE), Recall Cross-Entropy (RCE), and Recall Entropy (RE), which separately measure fidelity and two distinct aspects of diversity, inter- and intra-class. Our domain-agnostic metric, derived from the information-theoretic concepts of entropy and cross-entropy, can be dissected for both sample- and mode-level analysis. Our detailed experimental results demonstrate the sensitivity of our metric components to their respective qualities and reveal undesirable behaviors of other metrics.