# When Not to Classify: Anomaly Detection of Attacks (ADA) on DNN Classifiers at Test Time

arXiv: https://arxiv.org/abs/1712.06646 · HTML: https://mlsanity.com/paper/1712.06646/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2017.

## Built on by (most cited first, 9 of 9)

- [Adversarial Example Detection for DNN Models: A Review and Experimental Comparison](https://mlsanity.com/paper/2105.00203/) (2021)
- [Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them](https://mlsanity.com/paper/2107.11630/) (2021)
- [Adv-Bot: Realistic Adversarial Botnet Attacks against Network Intrusion Detection Systems](https://mlsanity.com/paper/2303.06664/) (2023)
- [TAD: Transfer Learning-based Multi-Adversarial Detection of Evasion Attacks against Network Intrusion Detection Systems](https://mlsanity.com/paper/2210.15700/) (2022)
- [Anomaly Detection of Adversarial Examples using Class-conditional Generative Adversarial Networks](https://mlsanity.com/paper/2105.10101/) (2021)
- [Backdoor Mitigation by Correcting the Distribution of Neural Activations](https://mlsanity.com/paper/2308.09850/) (2023)
- [MixDefense: A Defense-in-Depth Framework for Adversarial Example Detection Based on Statistical and Semantic Analysis](https://mlsanity.com/paper/2104.10076/) (2021)
- [A BIC-based Mixture Model Defense against Data Poisoning Attacks on Classifiers](https://mlsanity.com/paper/2105.13530/) (2021)
- [Detection of Adversarial Attacks in Robotic Perception](https://mlsanity.com/paper/2603.28594/) (2026)

## Abstract (arXiv)

A significant threat to the recent, wide deployment of machine learning-based systems, including deep neural networks (DNNs), is adversarial learning attacks. We analyze possible test-time evasion-attack mechanisms and show that, in some important cases, when the image has been attacked, correctly classifying it has no utility: i) when the image to be attacked is (even arbitrarily) selected from the attacker's cache; ii) when the sole recipient of the classifier's decision is the attacker. Moreover, in some application domains and scenarios it is highly actionable to detect the attack irrespective of correctly classifying in the face of it (with classification still performed if no attack is detected). We hypothesize that, even if human-imperceptible, adversarial perturbations are machine-detectable. We propose a purely unsupervised anomaly detector (AD) that, unlike previous works: i) models the joint density of a deep layer using highly suitable null hypothesis density models (matched in particular to the non- negative support for RELU layers); ii) exploits multiple DNN layers; iii) leverages a "source" and "destination" class concept, source class uncertainty, the class confusion matrix, and DNN weight information in constructing a novel decision statistic grounded in the Kullback-Leibler divergence. Tested on MNIST and CIFAR-10 image databases under three prominent attack strategies, our approach outperforms previous detection methods, achieving strong ROC AUC detection accuracy on two attacks and better accuracy than recently reported for a variety of methods on the strongest (CW) attack. We also evaluate a fully white box attack on our system. Finally, we evaluate other important performance measures, such as classification accuracy, versus detection rate and attack strength.