# Motivating the Rules of the Game for Adversarial Example Research

arXiv: https://arxiv.org/abs/1807.06732 · HTML: https://mlsanity.com/paper/1807.06732/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2018.

## Built on by (most cited first, 30 of 57)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning](https://mlsanity.com/paper/2205.01992/) (2022)
- [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023)
- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [Data Poisoning Won't Save You From Facial Recognition](https://mlsanity.com/paper/2106.14851/) (2021)
- [Towards Defending against Adversarial Examples via Attack-Invariant Features](https://mlsanity.com/paper/2106.05036/) (2021)
- [Isometric 3D Adversarial Examples in the Physical World](https://mlsanity.com/paper/2210.15291/) (2022)
- [Machine Learning Security in Industry: A Quantitative Survey](https://mlsanity.com/paper/2207.05164/) (2022)
- [A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against Adversarial Attacks](https://mlsanity.com/paper/2310.00633/) (2023)
- [Removing Adversarial Noise in Class Activation Feature Space](https://mlsanity.com/paper/2104.09197/) (2021)
- [Towards more Practical Threat Models in Artificial Intelligence Security](https://mlsanity.com/paper/2311.09994/) (2023)
- [Double Bubble, Toil and Trouble: Enhancing Certified Robustness through Transitivity](https://mlsanity.com/paper/2210.06077/) (2022)
- [Position: Adversarial ML for LLMs Is Not Making Any Progress](https://mlsanity.com/paper/2502.02260/) (2025)
- [Why adversarial training can hurt robust accuracy](https://mlsanity.com/paper/2203.02006/) (2022)
- [Trusted Artificial Intelligence: Towards Certification of Machine Learning Applications](https://mlsanity.com/paper/2103.16910/) (2021)
- [Evading Black-box Classifiers Without Breaking Eggs](https://mlsanity.com/paper/2306.02895/) (2023)
- [Token-Modification Adversarial Attacks for Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.00676/) (2021)
- [Exposing Previously Undetectable Faults in Deep Neural Networks](https://mlsanity.com/paper/2106.00576/) (2021)
- [Adversarial robustness of sparse local Lipschitz predictors](https://mlsanity.com/paper/2202.13216/) (2022)
- [Dompteur: Taming Audio Adversarial Examples](https://mlsanity.com/paper/2102.05431/) (2021)
- [Rethinking Natural Adversarial Examples for Classification Models](https://mlsanity.com/paper/2102.11731/) (2021)
- [It Is All About Data: A Survey on the Effects of Data on Adversarial Robustness](https://mlsanity.com/paper/2303.09767/) (2023)
- [Machine Learning Robustness: A Primer](https://mlsanity.com/paper/2404.00897/) (2024)
- [Domain Invariant Adversarial Learning](https://mlsanity.com/paper/2104.00322/) (2021)
- [Natural Adversarial Objects](https://mlsanity.com/paper/2111.04204/) (2021)
- [PubDef: Defending Against Transfer Attacks From Public Models](https://mlsanity.com/paper/2310.17645/) (2023)
- [σ-zero: Gradient-based Optimization of ℓ0-norm Adversarial Examples](https://mlsanity.com/paper/2402.01879/) (2024)
- [A Multiclass Boosting Framework for Achieving Fast and Provable Adversarial Robustness](https://mlsanity.com/paper/2103.01276/) (2021)
- [Selection of Source Images Heavily Influences the Effectiveness of Adversarial Attacks](https://mlsanity.com/paper/2106.07141/) (2021)

## Abstract (arXiv)

Advances in machine learning have led to broad deployment of systems with impressive performance on important problems. Nonetheless, these systems can be induced to make errors on data that are surprisingly similar to examples the learned system handles correctly. The existence of these errors raises a variety of questions about out-of-sample generalization and whether bad actors might use such examples to abuse deployed systems. As a result of these security concerns, there has been a flurry of recent papers proposing algorithms to defend against such malicious perturbations of correctly handled examples. It is unclear how such misclassifications represent a different kind of security problem than other errors, or even other attacker-produced examples that have no specific relationship to an uncorrupted input. In this paper, we argue that adversarial example defense papers have, to date, mostly considered abstract, toy games that do not relate to any specific security concern. Furthermore, defense papers have not yet precisely described all the abilities and limitations of attackers that would be relevant in practical security. Towards this end, we establish a taxonomy of motivations, constraints, and abilities for more plausible adversaries. Finally, we provide a series of recommendations outlining a path forward for future work to more clearly articulate the threat model and perform more meaningful evaluation.