# Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing

arXiv: https://arxiv.org/abs/1812.05793 · HTML: https://mlsanity.com/paper/1812.05793/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2018.

## Built on by (most cited first, 30 of 48)

- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [Out-Of-Distribution Detection Is Not All You Need](https://mlsanity.com/paper/2211.16158/) (2022)
- [LLMEffiChecker: Understanding and Testing Efficiency Degradation of Large Language Models](https://mlsanity.com/paper/2210.03696/) (2022)
- [BDMMT: Backdoor Sample Detection for Language Models through Model Mutation Testing](https://mlsanity.com/paper/2301.10412/) (2023)
- [AEON: A Method for Automatic Evaluation of NLP Test Cases](https://mlsanity.com/paper/2205.06439/) (2022)
- [Attack as Defense: Characterizing Adversarial Examples using Robustness](https://mlsanity.com/paper/2103.07633/) (2021)
- [Automatic Fairness Testing of Neural Classifiers through Adversarial Sampling](https://mlsanity.com/paper/2107.08176/) (2021)
- [Towards Practical Robustness Analysis for DNNs based on PAC-Model Learning](https://mlsanity.com/paper/2101.10102/) (2021)
- [DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score](https://mlsanity.com/paper/2109.07514/) (2021)
- [PatchCensor: Patch Robustness Certification for Transformers via Exhaustive Testing](https://mlsanity.com/paper/2111.10481/) (2021)
- [Unifying Evaluation of Machine Learning Safety Monitors](https://mlsanity.com/paper/2208.14660/) (2022)
- [Increasing the Confidence of Deep Neural Networks by Coverage Analysis](https://mlsanity.com/paper/2101.12100/) (2021)
- [Fairness Testing of Deep Image Classification with Adequacy Metrics](https://mlsanity.com/paper/2111.08856/) (2021)
- [Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forward](https://mlsanity.com/paper/2404.08517/) (2024)
- [Estimating Predictive Uncertainty Under Program Data Distribution Shift](https://mlsanity.com/paper/2107.10989/) (2021)
- [Verifying Neural Networks Against Backdoor Attacks](https://mlsanity.com/paper/2205.06992/) (2022)
- [COMET: Coverage-guided Model Generation For Deep Learning Library Testing](https://mlsanity.com/paper/2208.01508/) (2022)
- [Exploiting Machine Unlearning for Backdoor Attacks in Deep Learning System](https://mlsanity.com/paper/2310.10659/) (2023)
- [Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Study](https://mlsanity.com/paper/2402.05939/) (2024)
- [Safety Monitoring of Machine Learning Perception Functions: a Survey](https://mlsanity.com/paper/2412.06869/) (2024)
- [MUTEN: Boosting Gradient-Based Adversarial Attacks via Mutant-Based Ensembles](https://mlsanity.com/paper/2109.12838/) (2021)
- [Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques](https://mlsanity.com/paper/2202.12139/) (2022)
- [Input-specific Attention Subnetworks for Adversarial Detection](https://mlsanity.com/paper/2203.12298/) (2022)
- [Machine Translation Testing via Syntactic Tree Pruning](https://mlsanity.com/paper/2401.00751/) (2024)
- [QuanTest: Entanglement-Guided Testing of Quantum Neural Network Systems](https://mlsanity.com/paper/2402.12950/) (2024)
- [MILE: A Mutation Testing Framework of In-Context Learning Systems](https://mlsanity.com/paper/2409.04831/) (2024)
- [GenoArmory: A Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models](https://mlsanity.com/paper/2505.10983/) (2025)
- [Fingerprinting Image-to-Image Generative Adversarial Networks](https://mlsanity.com/paper/2106.11760/) (2021)
- [GGT: Graph-Guided Testing for Adversarial Sample Detection of Deep Neural Network](https://mlsanity.com/paper/2107.07043/) (2021)
- [Provably Valid and Diverse Mutations of Real-World Media Data for DNN Testing](https://mlsanity.com/paper/2112.01956/) (2021)

## Abstract (arXiv)

Deep neural networks (DNN) have been shown to be useful in a wide range of applications. However, they are also known to be vulnerable to adversarial samples. By transforming a normal sample with some carefully crafted human imperceptible perturbations, even highly accurate DNN make wrong decisions. Multiple defense mechanisms have been proposed which aim to hinder the generation of such adversarial samples. However, a recent work show that most of them are ineffective. In this work, we propose an alternative approach to detect adversarial samples at runtime. Our main observation is that adversarial samples are much more sensitive than normal samples if we impose random mutations on the DNN. We thus first propose a measure of `sensitivity' and show empirically that normal samples and adversarial samples have distinguishable sensitivity. We then integrate statistical hypothesis testing and model mutation testing to check whether an input sample is likely to be normal or adversarial at runtime by measuring its sensitivity. We evaluated our approach on the MNIST and CIFAR10 datasets. The results show that our approach detects adversarial samples generated by state-of-the-art attacking methods efficiently and accurately.