# Metrics and methods for robustness evaluation of neural networks with generative models

arXiv: https://arxiv.org/abs/2003.01993 · HTML: https://mlsanity.com/paper/2003.01993/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2020.

## Built on by (most cited first, 12 of 12)

- [Adversarial random forests for density estimation and generative modeling](https://mlsanity.com/paper/2205.09435/) (2022)
- [A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities](https://mlsanity.com/paper/2210.08906/) (2022)
- [LuciBot: Automated Robot Policy Learning from Generated Videos](https://mlsanity.com/paper/2503.09871/) (2025)
- [Agent Benchmarks Fail Public Sector Requirements](https://mlsanity.com/paper/2601.20617/) (2026)
- [Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space](https://mlsanity.com/paper/2106.12303/) (2021)
- [Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study](https://mlsanity.com/paper/2306.07737/) (2023)
- [Quantifying Robustness: A Benchmarking Framework for Deep Learning Forecasting in Cyber-Physical Systems](https://mlsanity.com/paper/2504.03494/) (2025)
- [Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric](https://mlsanity.com/paper/2410.17628/) (2024)
- [HyperNet-Adaptation for Diffusion-Based Test Case Generation](https://mlsanity.com/paper/2601.15041/) (2026)
- [When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models](https://mlsanity.com/paper/2603.27759/) (2026)
- [Generative Testing of Automated Speech Recognition Systems](https://mlsanity.com/paper/2607.09833/) (2026)
- [ARMOR: Manifold-Oriented Training for Adversarially Robust Aerial Object Detection under Data Scarcity](https://mlsanity.com/paper/2608.29510/) (2026)

## Abstract (arXiv)

Recent studies have shown that modern deep neural network classifiers are easy to fool, assuming that an adversary is able to slightly modify their inputs. Many papers have proposed adversarial attacks, defenses and methods to measure robustness to such adversarial perturbations. However, most commonly considered adversarial examples are based on ℓp-bounded perturbations in the input space of the neural network, which are unlikely to arise naturally. Recently, especially in computer vision, researchers discovered "natural" or "semantic" perturbations, such as rotations, changes of brightness, or more high-level changes, but these perturbations have not yet been systematically utilized to measure the performance of classifiers. In this paper, we propose several metrics to measure robustness of classifiers to natural adversarial examples, and methods to evaluate them. These metrics, called latent space performance metrics, are based on the ability of generative models to capture probability distributions, and are defined in their latent spaces. On three image classification case studies, we evaluate the proposed metrics for several classifiers, including ones trained in conventional and robust ways. We find that the latent counterparts of adversarial robustness are associated with the accuracy of the classifier rather than its conventional adversarial robustness, but the latter is still reflected on the properties of found latent perturbations. In addition, our novel method of finding latent adversarial perturbations demonstrates that these perturbations are often perceptually small.