# Reliable Robustness Evaluation via Automatically Constructed Attack Ensembles

arXiv: https://arxiv.org/abs/2211.12713 · HTML: https://mlsanity.com/paper/2211.12713/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 30 of 35)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack](https://mlsanity.com/paper/1907.02044/) (2019)
- [On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models](https://mlsanity.com/paper/1810.12715/) (2018)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples](https://mlsanity.com/paper/2010.03593/) (2020)
- [Fixing Data Augmentation to Improve Adversarial Robustness](https://mlsanity.com/paper/2103.01946/) (2021)
- [Bag of Tricks for Adversarial Training](https://mlsanity.com/paper/2010.00467/) (2020)
- [Geometry-aware Instance-reweighted Adversarial Training](https://mlsanity.com/paper/2010.01736/) (2020)
- [MMA Training: Direct Input Space Margin Maximization through Adversarial Training](https://mlsanity.com/paper/1812.02637/) (2018)
- [Are Labels Required for Improving Adversarial Robustness?](https://mlsanity.com/paper/1905.13725/) (2019)
- [Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?](https://mlsanity.com/paper/2104.09425/) (2021)
- [Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defenses](https://mlsanity.com/paper/1811.09600/) (2018)
- [Boosting Adversarial Training with Hypersphere Embedding](https://mlsanity.com/paper/2002.08619/) (2020)
- [Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training](https://mlsanity.com/paper/1907.10764/) (2019)
- [Learnable Boundary Guided Adversarial Training](https://mlsanity.com/paper/2011.11164/) (2020)
- [Adversarial Robustness on In- and Out-Distribution Improves Explainability](https://mlsanity.com/paper/2003.09461/) (2020)
- [Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective](https://mlsanity.com/paper/2009.03728/) (2020)
- [Provable Robustness of ReLU networks via Maximization of Linear Regions](https://mlsanity.com/paper/1810.07481/) (2018)
- [An Alternative Surrogate Loss for PGD-based Adversarial Testing](https://mlsanity.com/paper/1910.09338/) (2019)
- [Composite Adversarial Attacks](https://mlsanity.com/paper/2012.05434/) (2020)

## Built on by (most cited first, 6 of 6)

- [Exploring the Adversarial Frontier: Quantifying Robustness via Adversarial Hypervolume](https://mlsanity.com/paper/2403.05100/) (2024)
- [Memory efficient location recommendation through proximity-aware representation](https://mlsanity.com/paper/2310.06484/) (2023)
- [Enhancing Output Diversity Improves Conjugate Gradient-based Adversarial Attacks](https://mlsanity.com/paper/2408.03972/) (2024)
- [Computable Model-Independent Bounds for Adversarial Quantum Machine Learning](https://mlsanity.com/paper/2411.06863/) (2024)
- [Towards Provably Unlearnable Examples via Bayes Error Optimisation](https://mlsanity.com/paper/2511.08191/) (2025)
- [Adversarial Frontiers: Minimum-Norm Attack Ensembles for Robustness Evaluation](https://mlsanity.com/paper/2607.19855/) (2026)

## Abstract (arXiv)

Attack Ensemble (AE), which combines multiple attacks together, provides a reliable way to evaluate adversarial robustness. In practice, AEs are often constructed and tuned by human experts, which however tends to be sub-optimal and time-consuming. In this work, we present AutoAE, a conceptually simple approach for automatically constructing AEs. In brief, AutoAE repeatedly adds the attack and its iteration steps to the ensemble that maximizes ensemble improvement per additional iteration consumed. We show theoretically that AutoAE yields AEs provably within a constant factor of the optimal for a given defense. We then use AutoAE to construct two AEs for l∞ and l2 attacks, and apply them without any tuning or adaptation to 45 top adversarial defenses on the RobustBench leaderboard. In all except one cases we achieve equal or better (often the latter) robustness evaluation than existing AEs, and notably, in 29 cases we achieve better robustness evaluation than the best known one. Such performance of AutoAE shows itself as a reliable evaluation protocol for adversarial robustness, which further indicates the huge potential of automatic AE construction. Code is available at urlhttps://github.com/LeegerPENG/AutoAE.