# Accurate, reliable and fast robustness evaluation

arXiv: https://arxiv.org/abs/1907.01003 · HTML: https://mlsanity.com/paper/1907.01003/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2019.

## Built on by (most cited first, 30 of 43)

- [Robustness and Accuracy Could Be Reconcilable by (Proper) Definition](https://mlsanity.com/paper/2202.10103/) (2022)
- [Mind the box: l1-APGD for sparse adversarial attacks on image classifiers](https://mlsanity.com/paper/2103.01208/) (2021)
- [Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints](https://mlsanity.com/paper/2102.12827/) (2021)
- [A Unified Wasserstein Distributional Robustness Framework for Adversarial Training](https://mlsanity.com/paper/2202.13437/) (2022)
- [GradDiv: Adversarial Robustness of Randomized Neural Networks via Gradient Diversity Regularization](https://mlsanity.com/paper/2107.02425/) (2021)
- [Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attacks](https://mlsanity.com/paper/2105.10304/) (2021)
- [Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples](https://mlsanity.com/paper/2106.09947/) (2021)
- [Proximal Splitting Adversarial Attacks for Semantic Segmentation](https://mlsanity.com/paper/2206.07179/) (2022)
- [Unrestricted Black-box Adversarial Attack Using GAN with Limited Queries](https://mlsanity.com/paper/2208.11613/) (2022)
- [LLMEffiChecker: Understanding and Testing Efficiency Degradation of Large Language Models](https://mlsanity.com/paper/2210.03696/) (2022)
- [AttackBench: Evaluating Gradient-based Attacks for Adversarial Examples](https://mlsanity.com/paper/2404.19460/) (2024)
- [Identifying Untrustworthy Predictions in Neural Networks by Geometric Gradient Analysis](https://mlsanity.com/paper/2102.12196/) (2021)
- [Increasing Confidence in Adversarial Robustness Evaluations](https://mlsanity.com/paper/2206.13991/) (2022)
- [Visual Robustness Benchmark for Visual Question Answering (VQA)](https://mlsanity.com/paper/2407.03386/) (2024)
- [Global-Local Regularization Via Distributional Robustness](https://mlsanity.com/paper/2203.00553/) (2022)
- [Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness](https://mlsanity.com/paper/2409.01249/) (2024)
- [PDPGD: Primal-Dual Proximal Gradient Descent Adversarial Attack](https://mlsanity.com/paper/2106.01538/) (2021)
- [Performance Evaluation of Adversarial Attacks: Discrepancies and Solutions](https://mlsanity.com/paper/2104.11103/) (2021)
- [σ-zero: Gradient-based Optimization of ℓ0-norm Adversarial Examples](https://mlsanity.com/paper/2402.01879/) (2024)
- [Selection of Source Images Heavily Influences the Effectiveness of Adversarial Attacks](https://mlsanity.com/paper/2106.07141/) (2021)
- [Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs](https://mlsanity.com/paper/2110.10645/) (2021)
- [Towards Out-of-Distribution Adversarial Robustness](https://mlsanity.com/paper/2210.03150/) (2022)
- [Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction](https://mlsanity.com/paper/2301.03573/) (2023)
- [Users are the North Star for AI Transparency](https://mlsanity.com/paper/2303.05500/) (2023)
- [Pruning in the Face of Adversaries](https://mlsanity.com/paper/2108.08560/) (2021)
- [EG-Booster: Explanation-Guided Booster of ML Evasion Attacks](https://mlsanity.com/paper/2108.13930/) (2021)
- [Enhancing Robustness of Neural Networks through Fourier Stabilization](https://mlsanity.com/paper/2106.04435/) (2021)
- [PopSkipJump: Decision-Based Attack for Probabilistic Classifiers](https://mlsanity.com/paper/2106.07445/) (2021)
- [Testing robustness of predictions of trained classifiers against naturally occurring perturbations](https://mlsanity.com/paper/2204.10046/) (2022)
- [Generating Adversarial Examples with Task Oriented Multi-Objective Optimization](https://mlsanity.com/paper/2304.13229/) (2023)

## Abstract (arXiv)

Throughout the past five years, the susceptibility of neural networks to minimal adversarial perturbations has moved from a peculiar phenomenon to a core issue in Deep Learning. Despite much attention, however, progress towards more robust models is significantly impaired by the difficulty of evaluating the robustness of neural network models. Today's methods are either fast but brittle (gradient-based attacks), or they are fairly reliable but slow (score- and decision-based attacks). We here develop a new set of gradient-based adversarial attacks which (a) are more reliable in the face of gradient-masking than other gradient-based attacks, (b) perform better and are more query efficient than current state-of-the-art gradient-based attacks, (c) can be flexibly adapted to a wide range of adversarial criteria and (d) require virtually no hyperparameter tuning. These findings are carefully validated across a diverse set of six different models and hold for L0, L1, L2 and Linf in both targeted as well as untargeted scenarios. Implementations will soon be available in all major toolboxes (Foolbox, CleverHans and ART). We hope that this class of attacks will make robustness evaluations easier and more reliable, thus contributing to more signal in the search for more robust machine learning models.