# Intriguing properties of adversarial training at scale

arXiv: https://arxiv.org/abs/1906.03787 · HTML: https://mlsanity.com/paper/1906.03787/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 30 of 33)

- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [AugMax: Adversarial Composition of Random Augmentations for Robust Training](https://mlsanity.com/paper/2110.13771/) (2021)
- [Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks](https://mlsanity.com/paper/2110.03825/) (2021)
- [Exploring Memorization in Adversarial Training](https://mlsanity.com/paper/2106.01606/) (2021)
- [Rethinking "Batch" in BatchNorm](https://mlsanity.com/paper/2105.07576/) (2021)
- [A Comprehensive Evaluation Framework for Deep Model Robustness](https://mlsanity.com/paper/2101.09617/) (2021)
- [Removing Batch Normalization Boosts Adversarial Training](https://mlsanity.com/paper/2207.01156/) (2022)
- [On the Effect of Pruning on Adversarial Robustness](https://mlsanity.com/paper/2108.04890/) (2021)
- [Parameterizing Activation Functions for Adversarial Robustness](https://mlsanity.com/paper/2110.05626/) (2021)
- [Defensive Patches for Robust Recognition in the Physical World](https://mlsanity.com/paper/2204.06213/) (2022)
- [Meta Adversarial Training against Universal Patches](https://mlsanity.com/paper/2101.11453/) (2021)
- [Pure Noise to the Rescue of Insufficient Data: Improving Imbalanced Classification by Training on Random Noise Images](https://mlsanity.com/paper/2112.08810/) (2021)
- [Generative Adversarial Networks: A Survey Towards Private and Secure Applications](https://mlsanity.com/paper/2106.03785/) (2021)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [Understanding Robust Learning through the Lens of Representation Similarities](https://mlsanity.com/paper/2206.09868/) (2022)
- [MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps](https://mlsanity.com/paper/2111.05073/) (2021)
- [NoiLIn: Improving Adversarial Training and Correcting Stereotype of Noisy Labels](https://mlsanity.com/paper/2105.14676/) (2021)
- [Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity](https://mlsanity.com/paper/2102.11382/) (2021)
- [Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room](https://mlsanity.com/paper/2108.11801/) (2021)
- [Non-adversarial Robustness of Deep Learning Methods for Computer Vision](https://mlsanity.com/paper/2305.14986/) (2023)
- [Robustness in Compressed Neural Networks for Object Detection](https://mlsanity.com/paper/2102.05509/) (2021)
- [On Fragile Features and Batch Normalization in Adversarial Training](https://mlsanity.com/paper/2204.12393/) (2022)
- [Robust SAR ATR on MSTAR with Deep Learning Models trained on Full Synthetic MOCEM data](https://mlsanity.com/paper/2206.07352/) (2022)
- [Tiny Adversarial Mulit-Objective Oneshot Neural Architecture Search](https://mlsanity.com/paper/2103.00363/) (2021)
- [Towards Alternative Techniques for Improving Adversarial Robustness: Analysis of Adversarial Training at a Spectrum of Perturbations](https://mlsanity.com/paper/2206.06496/) (2022)
- [Differentiable Search of Accurate and Robust Architectures](https://mlsanity.com/paper/2212.14049/) (2022)
- [Towards Bridging the gap between Empirical and Certified Robustness against Adversarial Examples](https://mlsanity.com/paper/2102.05096/) (2021)
- [THAT: Two Head Adversarial Training for Improving Robustness at Scale](https://mlsanity.com/paper/2103.13612/) (2021)
- [Overparametrization improves robustness against adversarial attacks: A replication study](https://mlsanity.com/paper/2202.09735/) (2022)
- [A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness](https://mlsanity.com/paper/2204.00426/) (2022)

## Abstract (arXiv)

Adversarial training is one of the main defenses against adversarial attacks. In this paper, we provide the first rigorous study on diagnosing elements of adversarial training, which reveals two intriguing properties. First, we study the role of normalization. Batch normalization (BN) is a crucial element for achieving state-of-the-art performance on many vision tasks, but we show it may prevent networks from obtaining strong robustness in adversarial training. One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, i.e., 18.3%. We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains. This two-domain hypothesis may explain the issue of BN when training with a mixture of clean and adversarial images, as estimating normalization statistics of this mixture distribution is challenging. Guided by this two-domain hypothesis, we show disentangling the mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness. Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness. Second, we study the role of network capacity. We find our so-called "deep" networks are still shallow for the task of adversarial learning. Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to "deep" networks (e.g., ResNet-152), adversarial training exhibits a much stronger demand on deeper networks to achieve higher adversarial robustness. This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., ResNet-638.