# On the Robustness of Quality Measures for GANs

arXiv: https://arxiv.org/abs/2201.13019 · HTML: https://mlsanity.com/paper/2201.13019/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 22 of 22)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Effectively Unbiased FID and Inception Score and where to find them](https://mlsanity.com/paper/1911.07023/) (2019)
- [On the Robustness of Semantic Segmentation Models to Adversarial Attacks](https://mlsanity.com/paper/1711.09856/) (2017)
- [An empirical study on evaluation metrics of generative adversarial networks](https://mlsanity.com/paper/1806.07755/) (2018)
- [How good is my GAN?](https://mlsanity.com/paper/1807.09499/) (2018)
- [PointGuard: Provably Robust 3D Point Cloud Classification](https://mlsanity.com/paper/2103.03046/) (2021)
- [Adversarial Robustness of Flow-Based Generative Models](https://mlsanity.com/paper/1911.08654/) (2019)

## Built on by (most cited first, 11 of 11)

- [The Role of ImageNet Classes in Fr'echet Inception Distance](https://mlsanity.com/paper/2203.06026/) (2022)
- [HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models](https://mlsanity.com/paper/2304.05390/) (2023)
- [On the Content Bias in Fr'echet Video Distance](https://mlsanity.com/paper/2404.12391/) (2024)
- [Revisiting the Evaluation of Image Synthesis with GANs](https://mlsanity.com/paper/2304.01999/) (2023)
- [Generalizability of Adversarial Robustness Under Distribution Shifts](https://mlsanity.com/paper/2209.15042/) (2022)
- [Universal Adversarial Defense in Remote Sensing Based on Pre-trained Denoising Diffusion Models](https://mlsanity.com/paper/2307.16865/) (2023)
- [Adversarial Robustness in Unsupervised Machine Learning: A Systematic Review](https://mlsanity.com/paper/2306.00687/) (2023)
- [Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability](https://mlsanity.com/paper/2312.10634/) (2023)
- [Data Measurements for Decentralized Data Markets](https://mlsanity.com/paper/2406.04257/) (2024)
- [Counterfactual Explanations on Robust Perceptual Geodesics](https://mlsanity.com/paper/2601.18678/) (2026)
- [What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation](https://mlsanity.com/paper/2608.24881/) (2026)

## Abstract (arXiv)

This work evaluates the robustness of quality measures of generative models such as Inception Score (IS) and Fr'echet Inception Distance (FID). Analogous to the vulnerability of deep models against a variety of adversarial attacks, we show that such metrics can also be manipulated by additive pixel perturbations. Our experiments indicate that one can generate a distribution of images with very high scores but low perceptual quality. Conversely, one can optimize for small imperceptible perturbations that, when added to real world images, deteriorate their scores. We further extend our evaluation to generative models themselves, including the state of the art network StyleGANv2. We show the vulnerability of both the generative model and the FID against additive perturbations in the latent space. Finally, we show that the FID can be robustified by simply replacing the standard Inception with a robust Inception. We validate the effectiveness of the robustified metric through extensive experiments, showing it is more robust against manipulation.