# How many perturbations break this model? Evaluating robustness beyond adversarial accuracy

arXiv: https://arxiv.org/abs/2207.04129 · HTML: https://mlsanity.com/paper/2207.04129/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 48 references that are in the map.
- First posted on arXiv 8 Jul 2022.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models](https://mlsanity.com/paper/1805.06605/) (2018)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality](https://mlsanity.com/paper/1801.02613/) (2018)
- [Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples](https://mlsanity.com/paper/2010.03593/) (2020)
- [Fixing Data Augmentation to Improve Adversarial Robustness](https://mlsanity.com/paper/2103.01946/) (2021)
- [The Space of Transferable Adversarial Examples](https://mlsanity.com/paper/1704.03453/) (2017)

## Built on by (most cited first, 9 of 9)

- [The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?](https://mlsanity.com/paper/2412.03597/) (2024)
- [An Analytic Solution to Covariance Propagation in Neural Networks](https://mlsanity.com/paper/2403.16163/) (2024)
- [Exploring the Adversarial Frontier: Quantifying Robustness via Adversarial Hypervolume](https://mlsanity.com/paper/2403.05100/) (2024)
- [GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models](https://mlsanity.com/paper/2304.09875/) (2023)
- [Adversarial Robustness of VAEs across Intersectional Subgroups](https://mlsanity.com/paper/2407.03864/) (2024)
- [Quadratic Upper Bound for Boosting Robustness](https://mlsanity.com/paper/2601.13645/) (2026)
- [How Worst-Case Are Adversarial Attacks? Linking Adversarial and Perturbation Robustness](https://mlsanity.com/paper/2601.14519/) (2026)
- [Feature-Space Smoothing: Certified Robustness of Deep Representations](https://mlsanity.com/paper/2601.16200/) (2026)
- [Compression as an Adversarial Amplifier Through Decision Space Reduction](https://mlsanity.com/paper/2604.06954/) (2026)

## Abstract (arXiv)

Robustness to adversarial attacks is typically evaluated with adversarial accuracy. While essential, this metric does not capture all aspects of robustness and in particular leaves out the question of how many perturbations can be found for each point. In this work, we introduce an alternative approach, adversarial sparsity, which quantifies how difficult it is to find a successful perturbation given both an input point and a constraint on the direction of the perturbation. We show that sparsity provides valuable insight into neural networks in multiple ways: for instance, it illustrates important differences between current state-of-the-art robust models them that accuracy analysis does not, and suggests approaches for improving their robustness. When applying broken defenses effective against weak attacks but not strong ones, sparsity can discriminate between the totally ineffective and the partially effective defenses. Finally, with sparsity we can measure increases in robustness that do not affect accuracy: we show for example that data augmentation can by itself increase adversarial robustness, without using adversarial training.