# The Limitations of Deep Learning in Adversarial Settings

arXiv: https://arxiv.org/abs/1511.07528 · HTML: https://mlsanity.com/paper/1511.07528/

## Key facts

- Cited by 790 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2015.

## Built on by (most cited first, 30 of 790)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now](https://mlsanity.com/paper/2310.11868/) (2023)
- [Zero-Shot Machine Unlearning](https://mlsanity.com/paper/2201.05629/) (2022)
- [On the Convergence and Robustness of Adversarial Training](https://mlsanity.com/paper/2112.08304/) (2021)
- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Understanding Zero-Shot Adversarial Robustness for Large-Scale Models](https://mlsanity.com/paper/2212.07016/) (2022)
- [Globally-Robust Neural Networks](https://mlsanity.com/paper/2102.08452/) (2021)
- [Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks](https://mlsanity.com/paper/2106.09249/) (2021)
- [Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenon](https://mlsanity.com/paper/2203.03818/) (2022)
- [Adversarial Texture for Fooling Person Detectors in the Physical World](https://mlsanity.com/paper/2203.03373/) (2022)
- [Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attacks](https://mlsanity.com/paper/2110.12976/) (2021)
- [The effectiveness of feature attribution methods and its correlation with automatic evaluation scores](https://mlsanity.com/paper/2105.14944/) (2021)
- [Fast Adversarial Attacks on Language Models In One GPU Minute](https://mlsanity.com/paper/2402.15570/) (2024)
- [The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training](https://mlsanity.com/paper/2202.02643/) (2022)
- [Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability](https://mlsanity.com/paper/2111.10752/) (2021)
- [Practical Blind Membership Inference Attack via Differential Comparisons](https://mlsanity.com/paper/2101.01341/) (2021)
- [When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?](https://mlsanity.com/paper/2111.01124/) (2021)
- [Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity](https://mlsanity.com/paper/2203.05151/) (2022)
- [Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimization](https://mlsanity.com/paper/2112.12376/) (2021)
- [On Improving Adversarial Transferability of Vision Transformers](https://mlsanity.com/paper/2106.04169/) (2021)
- [BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning](https://mlsanity.com/paper/2303.14773/) (2023)
- [Towards Robustness Against Natural Language Word Substitutions](https://mlsanity.com/paper/2107.13541/) (2021)
- [How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses](https://mlsanity.com/paper/2305.10862/) (2023)
- [PatchCleanser: Certifiably Robust Defense against Adversarial Patches for Any Image Classifier](https://mlsanity.com/paper/2108.09135/) (2021)
- [X-Adv: Physical Adversarial Object Attacks against X-ray Prohibited Item Detection](https://mlsanity.com/paper/2302.09491/) (2023)
- [Relating Adversarially Robust Generalization to Flat Minima](https://mlsanity.com/paper/2104.04448/) (2021)
- [Physical Attack on Monocular Depth Estimation with Optimal Adversarial Patches](https://mlsanity.com/paper/2207.04718/) (2022)
- [Robust Semantic Communications with Masked VQ-VAE Enabled Codebook](https://mlsanity.com/paper/2206.04011/) (2022)
- [Don't Say No: Jailbreaking LLM by Suppressing Refusal](https://mlsanity.com/paper/2404.16369/) (2024)
- [Boosting Adversarial Transferability through Enhanced Momentum](https://mlsanity.com/paper/2103.10609/) (2021)

## Abstract (arXiv)

Deep learning takes advantage of large datasets and computationally efficient training algorithms to outperform other approaches at various machine learning tasks. However, imperfections in the training phase of deep neural networks make them vulnerable to adversarial samples: inputs crafted by adversaries with the intent of causing deep neural networks to misclassify. In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs. In an application to computer vision, we show that our algorithms can reliably produce samples correctly classified by human subjects but misclassified in specific targets by a DNN with a 97% adversarial success rate while only modifying on average 4.02% of the input features per sample. We then evaluate the vulnerability of different sample classes to adversarial perturbations by defining a hardness measure. Finally, we describe preliminary work outlining defenses against adversarial samples by defining a predictive measure of distance between a benign input and a target classification.