# Rarity Score : A New Metric to Evaluate the Uncommonness of Synthesized Images

arXiv: https://arxiv.org/abs/2206.08549 · HTML: https://mlsanity.com/paper/2206.08549/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [On the "steerability" of generative adversarial networks](https://mlsanity.com/paper/1907.07171/) (2019)
- [U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation](https://mlsanity.com/paper/1907.10830/) (2019)
- [Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks](https://mlsanity.com/paper/2202.10571/) (2022)
- [A Good Image Generator Is What You Need for High-Resolution Video Synthesis](https://mlsanity.com/paper/2104.15069/) (2021)
- [Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editing](https://mlsanity.com/paper/2104.14754/) (2021)
- [Tag2Pix: Line Art Colorization Using Text Tag With SECat and Changing Loss](https://mlsanity.com/paper/1908.05840/) (2019)
- [Feature Statistics Mixing Regularization for Generative Adversarial Networks](https://mlsanity.com/paper/2112.04120/) (2021)

## Built on by (most cited first, 30 of 35)

- [Don't Play Favorites: Minority Guidance for Diffusion Models](https://mlsanity.com/paper/2301.12334/) (2023)
- [Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans](https://mlsanity.com/paper/2310.19427/) (2023)
- [Towards a Scalable Reference-Free Evaluation of Generative Models](https://mlsanity.com/paper/2407.02961/) (2024)
- [Generative Uncertainty in Diffusion Models](https://mlsanity.com/paper/2502.20946/) (2025)
- [Self-Guided Generation of Minority Samples Using Diffusion Models](https://mlsanity.com/paper/2407.11555/) (2024)
- [Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models](https://mlsanity.com/paper/2306.01322/) (2023)
- [Minority-Focused Text-to-Image Generation via Prompt Optimization](https://mlsanity.com/paper/2410.07838/) (2024)
- [An Interpretable Evaluation of Entropy-based Novelty of Generative Models](https://mlsanity.com/paper/2402.17287/) (2024)
- [Unveiling Differences in Generative Models: A Scalable Differential Clustering Approach](https://mlsanity.com/paper/2405.02700/) (2024)
- [TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models](https://mlsanity.com/paper/2306.08013/) (2023)
- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [Be More Diverse than the Most Diverse: Optimal Mixtures of Generative Models via Mixture-UCB Bandit Algorithms](https://mlsanity.com/paper/2412.17622/) (2024)
- [A Multi-Armed Bandit Approach to Online Selection and Evaluation of Generative Models](https://mlsanity.com/paper/2406.07451/) (2024)
- [UMat: Uncertainty-Aware Single Image High Resolution Material Capture](https://mlsanity.com/paper/2305.16312/) (2023)
- [Consistent Distributed Ranking of Generative Models via Kernel Distances](https://mlsanity.com/paper/2310.11714/) (2023)
- [Forte : Finding Outliers with Representation Typicality Estimation](https://mlsanity.com/paper/2410.01322/) (2024)
- [Assessing Sample Quality via the Latent Space of Generative Models](https://mlsanity.com/paper/2407.15171/) (2024)
- [Visual DNA: Representing and Comparing Images using Distributions of Neuron Activations](https://mlsanity.com/paper/2304.10036/) (2023)
- [Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs](https://mlsanity.com/paper/2411.02817/) (2024)
- [Boost-and-Skip: A Simple Guidance-Free Diffusion for Minority Generation](https://mlsanity.com/paper/2502.06516/) (2025)
- [Attribute Based Interpretable Evaluation Metrics for Generative Models](https://mlsanity.com/paper/2310.17261/) (2023)
- [Improving the Effectiveness of Deep Generative Data](https://mlsanity.com/paper/2311.03959/) (2023)
- [Exploring the Impact of Synthetic Data for Aerial-view Human Detection](https://mlsanity.com/paper/2405.15203/) (2024)
- [Controlling the Fidelity and Diversity of Deep Generative Models via Pseudo Density](https://mlsanity.com/paper/2407.08659/) (2024)
- [Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability](https://mlsanity.com/paper/2312.10634/) (2023)
- [Uncertainty-aware Evaluation of Auxiliary Anomalies with the Expected Anomaly Posterior](https://mlsanity.com/paper/2405.13699/) (2024)
- [Exposing Diversity Bias in Deep Generative Models: Statistical Origins and Correction of Diversity Error](https://mlsanity.com/paper/2602.14682/) (2026)
- [ChatGPT4PCG 2 Competition: Prompt Engineering for Science Birds Level Generation](https://mlsanity.com/paper/2403.02610/) (2024)
- [Mixture-Greedy for Online Generative Model Selection: Is UCB Necessary in Diversity-Aware Multi-Armed Bandits?](https://mlsanity.com/paper/2603.21716/) (2026)
- [Beyond Generative Priors: Minority Sampling with JEPA-Guided Diffusion](https://mlsanity.com/paper/2605.24631/) (2026)

## Abstract (arXiv)

Evaluation metrics in image synthesis play a key role to measure performances of generative models. However, most metrics mainly focus on image fidelity. Existing diversity metrics are derived by comparing distributions, and thus they cannot quantify the diversity or rarity degree of each generated image. In this work, we propose a new evaluation metric, called `rarity score', to measure the individual rarity of each image synthesized by generative models. We first show empirical observation that common samples are close to each other and rare samples are far from each other in nearest-neighbor distances of feature space. We then use our metric to demonstrate that the extent to which different generative models produce rare images can be effectively compared. We also propose a method to compare rarities between datasets that share the same concept such as CelebA-HQ and FFHQ. Finally, we analyze the use of metrics in different designs of feature spaces to better understand the relationship between feature spaces and resulting sparse images. Code will be publicly available online for the research community.