# The Limitations of Adversarial Training and the Blind-Spot Attack

arXiv: https://arxiv.org/abs/1901.04684 · HTML: https://mlsanity.com/paper/1901.04684/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2019.

## Built on by (most cited first, 30 of 63)

- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Models](https://mlsanity.com/paper/2408.16807/) (2024)
- [The Threat of Offensive AI to Organizations](https://mlsanity.com/paper/2106.15764/) (2021)
- [ADBM: Adversarial diffusion bridge model for reliable adversarial purification](https://mlsanity.com/paper/2408.00315/) (2024)
- [Exploring the Relationship between Architecture and Adversarially Robust Generalization](https://mlsanity.com/paper/2209.14105/) (2022)
- [A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against Adversarial Attacks](https://mlsanity.com/paper/2310.00633/) (2023)
- [Recognizing Object by Components with Human Prior Knowledge Enhances Adversarial Robustness of Deep Neural Networks](https://mlsanity.com/paper/2212.01806/) (2022)
- [An Analysis of Recent Advances in Deepfake Image Detection in an Evolving Threat Landscape](https://mlsanity.com/paper/2404.16212/) (2024)
- [Adversarial Visual Robustness by Causal Intervention](https://mlsanity.com/paper/2106.09534/) (2021)
- [Improving Robustness by Enhancing Weak Subnets](https://mlsanity.com/paper/2201.12765/) (2022)
- [Local Competition and Stochasticity for Adversarial Robustness in Deep Learning](https://mlsanity.com/paper/2101.01121/) (2021)
- [Stochastic Local Winner-Takes-All Networks Enable Profound Adversarial Robustness](https://mlsanity.com/paper/2112.02671/) (2021)
- [Attacking Perceptual Similarity Metrics](https://mlsanity.com/paper/2305.08840/) (2023)
- [Does Typological Blinding Impede Cross-Lingual Sharing?](https://mlsanity.com/paper/2101.11888/) (2021)
- [Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation](https://mlsanity.com/paper/2104.05232/) (2021)
- [It Is All About Data: A Survey on the Effects of Data on Adversarial Robustness](https://mlsanity.com/paper/2303.09767/) (2023)
- [Learnability Lock: Authorized Learnability Control Through Adversarial Invertible Transformations](https://mlsanity.com/paper/2202.03576/) (2022)
- [Certified Robust Accuracy of Neural Networks Are Bounded due to Bayes Errors](https://mlsanity.com/paper/2405.11547/) (2024)
- [Evaluating the Robustness of Bayesian Neural Networks Against Different Types of Attacks](https://mlsanity.com/paper/2106.09223/) (2021)
- [Adversarially Robust Prototypical Few-shot Segmentation with Neural-ODEs](https://mlsanity.com/paper/2210.03429/) (2022)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [Learn2Weight: Parameter Adaptation against Similar-domain Adversarial Attacks](https://mlsanity.com/paper/2205.07315/) (2022)
- [Robust Prototypical Few-Shot Organ Segmentation with Regularized Neural-ODEs](https://mlsanity.com/paper/2208.12428/) (2022)
- [Auditing AI models for Verified Deployment under Semantic Specifications](https://mlsanity.com/paper/2109.12456/) (2021)
- [Adversarial Attacks and Defenses for Social Network Text Processing Applications: Techniques, Challenges and Future Research Directions](https://mlsanity.com/paper/2110.13980/) (2021)
- [Masked Spatial-Spectral Autoencoders Are Excellent Hyperspectral Defenders](https://mlsanity.com/paper/2207.07803/) (2022)
- [Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning](https://mlsanity.com/paper/2403.07078/) (2024)
- [Logit Calibration and Feature Contrast for Robust Federated Learning on Non-IID Data](https://mlsanity.com/paper/2404.06776/) (2024)
- [Mal2GCN: A Robust Malware Detection Approach Using Deep Graph Convolutional Networks With Non-Negative Weights](https://mlsanity.com/paper/2108.12473/) (2021)
- [Privacy-Enhancing Optical Embeddings for Lensless Classification](https://mlsanity.com/paper/2211.12864/) (2022)

## Abstract (arXiv)

The adversarial training procedure proposed by Madry et al. (2018) is one of the most effective methods to defend against adversarial examples in deep neural networks (DNNs). In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiveness (robustness on test set) of adversarial training has a strong correlation with the distance between a test point and the manifold of training data embedded by the network. Test examples that are relatively far away from this manifold are more likely to be vulnerable to adversarial attacks. Consequentially, an adversarial training based defense is susceptible to a new class of attacks, the "blind-spot attack", where the input images reside in "blind-spots" (low density regions) of the empirical distribution of training data but is still on the ground-truth data manifold. For MNIST, we found that these blind-spots can be easily found by simply scaling and shifting image pixel values. Most importantly, for large datasets with high dimensional and complex data manifold (CIFAR, ImageNet, etc), the existence of blind-spots in adversarial training makes defending on any valid test examples difficult due to the curse of dimensionality and the scarcity of training data. Additionally, we find that blind-spots also exist on provable defenses including (Wong & Kolter, 2018) and (Sinha et al., 2018) because these trainable robustness certificates can only be practically optimized on a limited set of training data.