# MEAD: A Multi-Armed Approach for Evaluation of Adversarial Examples Detectors

arXiv: https://arxiv.org/abs/2206.15415 · HTML: https://mlsanity.com/paper/2206.15415/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 30 Jun 2022.

## Builds on (most cited first, 28 of 28)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks](https://mlsanity.com/paper/1704.01155/) (2017)
- [Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack](https://mlsanity.com/paper/1907.02044/) (2019)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality](https://mlsanity.com/paper/1801.02613/) (2018)
- [MagNet: a Two-Pronged Defense against Adversarial Examples](https://mlsanity.com/paper/1705.09064/) (2017)
- [Detecting Adversarial Samples from Artifacts](https://mlsanity.com/paper/1703.00410/) (2017)
- [On Detecting Adversarial Perturbations](https://mlsanity.com/paper/1702.04267/) (2017)
- [On the (Statistical) Detection of Adversarial Examples](https://mlsanity.com/paper/1702.06280/) (2017)
- [Improving the Robustness of Deep Neural Networks via Stability Training](https://mlsanity.com/paper/1604.04326/) (2016)
- [Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics](https://mlsanity.com/paper/1612.07767/) (2016)
- [SafetyNet: Detecting and Rejecting Adversarial Examples Robustly](https://mlsanity.com/paper/1704.00103/) (2017)
- [DOCTOR: A Simple Method for Detecting Misclassification Errors](https://mlsanity.com/paper/2106.02395/) (2021)
- [Adversarial Example Detection for DNN Models: A Review and Experimental Comparison](https://mlsanity.com/paper/2105.00203/) (2021)
- [Detecting Adversarial Image Examples in Deep Networks with Adaptive Noise Reduction](https://mlsanity.com/paper/1705.08378/) (2017)
- [Attacks Meet Interpretability: Attribute-steered Detection of Adversarial Samples](https://mlsanity.com/paper/1810.11580/) (2018)
- [Deep Neural Rejection against Adversarial Examples](https://mlsanity.com/paper/1910.00470/) (2019)
- [Distributionally Adversarial Attack](https://mlsanity.com/paper/1808.05537/) (2018)
- [Revisiting Model's Uncertainty and Confidences for Adversarial Example Detection](https://mlsanity.com/paper/2103.05354/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Minimax Approach Against Multi-Armed Adversarial Attacks Detection](https://mlsanity.com/paper/2302.02216/) (2023)
- [Optimal Zero-Shot Detector for Multi-Armed Attacks](https://mlsanity.com/paper/2402.15808/) (2024)

## Abstract (arXiv)

Detection of adversarial examples has been a hot topic in the last years due to its importance for safely deploying machine learning algorithms in critical applications. However, the detection methods are generally validated by assuming a single implicitly known attack strategy, which does not necessarily account for real-life threats. Indeed, this can lead to an overoptimistic assessment of the detectors' performance and may induce some bias in the comparison between competing detection schemes. We propose a novel multi-armed framework, called MEAD, for evaluating detectors based on several attack strategies to overcome this limitation. Among them, we make use of three new objectives to generate attacks. The proposed performance metric is based on the worst-case scenario: detection is successful if and only if all different attacks are correctly recognized. Empirically, we show the effectiveness of our approach. Moreover, the poor performance obtained for state-of-the-art detectors opens a new exciting line of research.