# Stateful Detection of Black-Box Adversarial Attacks

arXiv: https://arxiv.org/abs/1907.05587 · HTML: https://mlsanity.com/paper/1907.05587/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2019.

## Built on by (most cited first, 30 of 56)

- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [AEVA: Black-box Backdoor Detection Using Adversarial Extreme Value Analysis](https://mlsanity.com/paper/2110.14880/) (2021)
- [PAL: Proxy-Guided Black-Box Attack on Large Language Models](https://mlsanity.com/paper/2402.09674/) (2024)
- [Random Noise Defense Against Query-Based Black-Box Attacks](https://mlsanity.com/paper/2104.11470/) (2021)
- [Privacy Side Channels in Machine Learning Systems](https://mlsanity.com/paper/2309.05610/) (2023)
- [Deepfake Text Detection: Limitations and Opportunities](https://mlsanity.com/paper/2210.09421/) (2022)
- [Energy-Latency Attacks via Sponge Poisoning](https://mlsanity.com/paper/2203.08147/) (2022)
- [Fooling the Eyes of Autonomous Vehicles: Robust Physical Adversarial Examples Against Traffic Sign Recognition Systems](https://mlsanity.com/paper/2201.06192/) (2022)
- [How to Steer Your Adversary: Targeted and Efficient Model Stealing Defenses with Gradient Redirection](https://mlsanity.com/paper/2206.14157/) (2022)
- [X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP](https://mlsanity.com/paper/2505.05528/) (2025)
- [Adversarial Attacks against Windows PE Malware Detection: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2112.12310/) (2021)
- [On the Effectiveness of Small Input Noise for Defending Against Query-based Black-Box Attacks](https://mlsanity.com/paper/2101.04829/) (2021)
- [Towards Good Practices in Evaluating Transfer Adversarial Attacks](https://mlsanity.com/paper/2211.09565/) (2022)
- [Defenses in Adversarial Machine Learning: A Survey](https://mlsanity.com/paper/2312.08890/) (2023)
- [Evading Black-box Classifiers Without Breaking Eggs](https://mlsanity.com/paper/2306.02895/) (2023)
- [Stateful Defenses for Machine Learning Models Are Not Yet Secure Against Black-box Attacks](https://mlsanity.com/paper/2303.06280/) (2023)
- [Does Typological Blinding Impede Cross-Lingual Sharing?](https://mlsanity.com/paper/2101.11888/) (2021)
- [MalProtect: Stateful Defense Against Adversarial Query Attacks in ML-based Malware Detection](https://mlsanity.com/paper/2302.10739/) (2023)
- [Benchmarking Misuse Mitigation Against Covert Adversaries](https://mlsanity.com/paper/2506.06414/) (2025)
- [PubDef: Defending Against Transfer Attacks From Public Models](https://mlsanity.com/paper/2310.17645/) (2023)
- [The Impact of Uniform Inputs on Activation Sparsity and Energy-Latency Attacks in Computer Vision](https://mlsanity.com/paper/2403.18587/) (2024)
- [AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuning](https://mlsanity.com/paper/2408.01978/) (2024)
- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [Black-box Optimization of LLM Outputs by Asking for Directions](https://mlsanity.com/paper/2510.16794/) (2025)
- [SoK: Pitfalls in Evaluating Black-Box Attacks](https://mlsanity.com/paper/2310.17534/) (2023)
- [Certifiable Black-Box Attacks with Randomized Adversarial Examples: Breaking Defenses with Provable Confidence](https://mlsanity.com/paper/2304.04343/) (2023)
- [The Adaptive Arms Race: Redefining Robustness in AI Security](https://mlsanity.com/paper/2312.13435/) (2023)
- [SoK: A Modularized Approach to Study the Security of Automatic Speech Recognition Systems](https://mlsanity.com/paper/2103.10651/) (2021)
- [Audit and Improve Robustness of Private Neural Networks on Encrypted Data](https://mlsanity.com/paper/2209.09996/) (2022)

## Abstract (arXiv)

The problem of adversarial examples, evasion attacks on machine learning classifiers, has proven extremely difficult to solve. This is true even when, as is the case in many practical settings, the classifier is hosted as a remote service and so the adversary does not have direct access to the model parameters. This paper argues that in such settings, defenders have a much larger space of actions than have been previously explored. Specifically, we deviate from the implicit assumption made by prior work that a defense must be a stateless function that operates on individual examples, and explore the possibility for stateful defenses. To begin, we develop a defense designed to detect the process of adversarial example generation. By keeping a history of the past queries, a defender can try to identify when a sequence of queries appears to be for the purpose of generating an adversarial example. We then introduce query blinding, a new class of attacks designed to bypass defenses that rely on such a defense approach. We believe that expanding the study of adversarial examples from stateless classifiers to stateful systems is not only more realistic for many black-box settings, but also gives the defender a much-needed advantage in responding to the adversary.