# AEVA: Black-box Backdoor Detection Using Adversarial Extreme Value Analysis

arXiv: https://arxiv.org/abs/2110.14880 · HTML: https://mlsanity.com/paper/2110.14880/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 45 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Invisible Backdoor Attack with Sample-Specific Triggers](https://mlsanity.com/paper/2012.03816/) (2020)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs](https://mlsanity.com/paper/1906.10842/) (2019)
- [TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems](https://mlsanity.com/paper/1908.01763/) (2019)
- [Sign-OPT: A Query-Efficient Hard-label Adversarial Attack](https://mlsanity.com/paper/1909.10773/) (2019)
- [QEBA: Query-Efficient Boundary-Based Blackbox Attack](https://mlsanity.com/paper/2005.14137/) (2020)
- [Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Cases](https://mlsanity.com/paper/2007.15802/) (2020)
- [Black-box Detection of Backdoor Attacks with Limited Information and Data](https://mlsanity.com/paper/2103.13127/) (2021)
- [Backdoor Attack against Speaker Verification](https://mlsanity.com/paper/2010.11607/) (2020)
- [Stateful Detection of Black-Box Adversarial Attacks](https://mlsanity.com/paper/1907.05587/) (2019)
- [Model-Reuse Attacks on Deep Learning Systems](https://mlsanity.com/paper/1812.00483/) (2018)

## Built on by (most cited first, 30 of 53)

- [Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning](https://mlsanity.com/paper/2205.01992/) (2022)
- [Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection](https://mlsanity.com/paper/2210.00875/) (2022)
- [Domain Watermark: Effective and Harmless Dataset Copyright Protection is Closed at Hand](https://mlsanity.com/paper/2310.14942/) (2023)
- [Detecting Backdoors During the Inference Stage Based on Corruption Robustness Consistency](https://mlsanity.com/paper/2303.18191/) (2023)
- [Black-box Backdoor Defense via Zero-shot Image Purification](https://mlsanity.com/paper/2303.12175/) (2023)
- [Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples](https://mlsanity.com/paper/2307.10562/) (2023)
- [Pre-trained Trojan Attacks for Visual Recognition](https://mlsanity.com/paper/2312.15172/) (2023)
- [Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features](https://mlsanity.com/paper/2306.16697/) (2023)
- [Few-Shot Backdoor Attacks on Visual Object Tracking](https://mlsanity.com/paper/2201.13178/) (2022)
- [Sibling-Attack: Rethinking Transferable Adversarial Attacks against Face Recognition](https://mlsanity.com/paper/2303.12512/) (2023)
- [BackdoorBox: A Python Toolbox for Backdoor Learning](https://mlsanity.com/paper/2302.01762/) (2023)
- [Distilling Cognitive Backdoor Patterns within an Image](https://mlsanity.com/paper/2301.10908/) (2023)
- [PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learning](https://mlsanity.com/paper/2202.03609/) (2022)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2407.19845/) (2024)
- [Universal Backdoor Attacks Detection via Adaptive Adversarial Probe](https://mlsanity.com/paper/2209.05244/) (2022)
- [Defenses in Adversarial Machine Learning: A Survey](https://mlsanity.com/paper/2312.08890/) (2023)
- [UFID: A Unified Framework for Input-level Backdoor Detection on Diffusion Models](https://mlsanity.com/paper/2404.01101/) (2024)
- [Test-Time Backdoor Detection for Object Detection Models](https://mlsanity.com/paper/2503.15293/) (2025)
- [DEJA VU: Continual Model Generalization For Unseen Domains](https://mlsanity.com/paper/2301.10418/) (2023)
- [Detection and Defense of Unlearnable Examples](https://mlsanity.com/paper/2312.08898/) (2023)
- [Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetwork](https://mlsanity.com/paper/2210.06428/) (2022)
- [Backdoor Defense via Suppressing Model Shortcuts](https://mlsanity.com/paper/2211.05631/) (2022)
- [MASTERKEY: Practical Backdoor Attack Against Speaker Verification Systems](https://mlsanity.com/paper/2309.06981/) (2023)
- [BAN: Detecting Backdoors Activated by Adversarial Neuron Noise](https://mlsanity.com/paper/2405.19928/) (2024)
- [XGBD: Explanation-Guided Graph Backdoor Detection](https://mlsanity.com/paper/2308.04406/) (2023)
- [Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness](https://mlsanity.com/paper/2405.20291/) (2024)
- [Towards a Defense Against Federated Backdoor Attacks Under Continuous Training](https://mlsanity.com/paper/2205.11736/) (2022)
- [A Proxy Attack-Free Strategy for Practically Improving the Poisoning Efficiency in Backdoor Attacks](https://mlsanity.com/paper/2306.08313/) (2023)
- [PhantomSound: Black-Box, Query-Efficient Audio Adversarial Attack via Split-Second Phoneme Injection](https://mlsanity.com/paper/2309.06960/) (2023)
- [Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers](https://mlsanity.com/paper/2405.10612/) (2024)

## Abstract (arXiv)

Deep neural networks (DNNs) are proved to be vulnerable against backdoor attacks. A backdoor is often embedded in the target DNNs through injecting a backdoor trigger into training examples, which can cause the target DNNs misclassify an input attached with the backdoor trigger. Existing backdoor detection methods often require the access to the original poisoned training data, the parameters of the target DNNs, or the predictive confidence for each given input, which are impractical in many real-world applications, e.g., on-device deployed DNNs. We address the black-box hard-label backdoor detection problem where the DNN is fully black-box and only its final output label is accessible. We approach this problem from the optimization perspective and show that the objective of backdoor detection is bounded by an adversarial objective. Further theoretical and empirical studies reveal that this adversarial objective leads to a solution with highly skewed distribution; a singularity is often observed in the adversarial map of a backdoor-infected example, which we call the adversarial singularity phenomenon. Based on this observation, we propose the adversarial extreme value analysis(AEVA) to detect backdoors in black-box neural networks. AEVA is based on an extreme value analysis of the adversarial map, computed from the monte-carlo gradient estimation. Evidenced by extensive experiments across multiple popular tasks and backdoor attacks, our approach is shown effective in detecting backdoor attacks under the black-box hard-label scenarios.