# Catastrophic overfitting can be induced with discriminative non-robust features

arXiv: https://arxiv.org/abs/2206.08242 · HTML: https://mlsanity.com/paper/2206.08242/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 30 of 33)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Towards Fast Computation of Certified Robustness for ReLU Networks](https://mlsanity.com/paper/1804.09699/) (2018)
- [Understanding and Improving Fast Adversarial Training](https://mlsanity.com/paper/2007.02617/) (2020)
- [Robustness via curvature regularization, and vice versa](https://mlsanity.com/paper/1811.09716/) (2018)
- [Adversarial Robustness through Local Linearization](https://mlsanity.com/paper/1907.02610/) (2019)
- [Understanding Catastrophic Overfitting in Single-step Adversarial Training](https://mlsanity.com/paper/2010.01799/) (2020)
- [On the effectiveness of adversarial training against common corruptions](https://mlsanity.com/paper/2103.02325/) (2021)
- [Make Some Noise: Reliable and Efficient Single-Step Adversarial Training](https://mlsanity.com/paper/2202.01181/) (2022)
- [How benign is benign overfitting?](https://mlsanity.com/paper/2007.04028/) (2020)
- [Towards Understanding Fast Adversarial Training](https://mlsanity.com/paper/2006.03089/) (2020)
- [Hold me tight! Influence of discriminative features on deep network boundaries](https://mlsanity.com/paper/2002.06349/) (2020)
- [Optimism in the Face of Adversity: Understanding and Improving Deep Learning through Adversarial Robustness](https://mlsanity.com/paper/2010.09624/) (2020)
- [FrequencyLowCut Pooling -- Plug & Play against Catastrophic Overfitting](https://mlsanity.com/paper/2204.00491/) (2022)
- [With Friends Like These, Who Needs Adversaries?](https://mlsanity.com/paper/1807.04200/) (2018)
- [Neural Anisotropy Directions](https://mlsanity.com/paper/2006.09717/) (2020)
- [ZeroGrad : Mitigating and Explaining Catastrophic Overfitting in FGSM Adversarial Training](https://mlsanity.com/paper/2103.15476/) (2021)

## Built on by (most cited first, 2 of 2)

- [Efficient local linearity regularization to overcome catastrophic overfitting](https://mlsanity.com/paper/2401.11618/) (2024)
- [On Using Certified Training towards Empirical Robustness](https://mlsanity.com/paper/2410.01617/) (2024)

## Abstract (arXiv)

Adversarial training (AT) is the de facto method for building robust neural networks, but it can be computationally expensive. To mitigate this, fast single-step attacks can be used, but this may lead to catastrophic overfitting (CO). This phenomenon appears when networks gain non-trivial robustness during the first stages of AT, but then reach a breaking point where they become vulnerable in just a few iterations. The mechanisms that lead to this failure mode are still poorly understood. In this work, we study the onset of CO in single-step AT methods through controlled modifications of typical datasets of natural images. In particular, we show that CO can be induced at much smaller ε values than it was observed before just by injecting images with seemingly innocuous features. These features aid non-robust classification but are not enough to achieve robustness on their own. Through extensive experiments we analyze this novel phenomenon and discover that the presence of these easy features induces a learning shortcut that leads to CO. Our findings provide new insights into the mechanisms of CO and improve our understanding of the dynamics of AT. The code to reproduce our experiments can be found at https://github.com/gortizji/cofeatures.