# Understanding and Mitigating the Tradeoff Between Robustness and Accuracy

arXiv: https://arxiv.org/abs/2002.10716 · HTML: https://mlsanity.com/paper/2002.10716/

## Key facts

- Cited by 149 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2020.

## Built on by (most cited first, 30 of 149)

- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Robustness and Accuracy Could Be Reconcilable by (Proper) Definition](https://mlsanity.com/paper/2202.10103/) (2022)
- [Improving Vision Transformers by Revisiting High-frequency Components](https://mlsanity.com/paper/2204.00993/) (2022)
- [Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks](https://mlsanity.com/paper/2110.03825/) (2021)
- [The Effects of Regularization and Data Augmentation are Class Dependent](https://mlsanity.com/paper/2204.03632/) (2022)
- [ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets](https://mlsanity.com/paper/2209.00613/) (2022)
- [Generate, Annotate, and Learn: NLP with Synthetic Text](https://mlsanity.com/paper/2106.06168/) (2021)
- [PRIME: A few primitives can boost robustness to common corruptions](https://mlsanity.com/paper/2112.13547/) (2021)
- [Adversarial Robustness with Semi-Infinite Constrained Learning](https://mlsanity.com/paper/2110.15767/) (2021)
- [Pyramid Adversarial Training Improves ViT Performance](https://mlsanity.com/paper/2111.15121/) (2021)
- [Among Us: Adversarially Robust Collaborative Perception by Consensus](https://mlsanity.com/paper/2303.09495/) (2023)
- [Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift](https://mlsanity.com/paper/2207.08977/) (2022)
- [Randomized Adversarial Training via Taylor Expansion](https://mlsanity.com/paper/2303.10653/) (2023)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [Knowledge Enhanced Machine Learning Pipeline against Diverse Adversarial Attacks](https://mlsanity.com/paper/2106.06235/) (2021)
- [Zero-Shot Learning by Harnessing Adversarial Samples](https://mlsanity.com/paper/2308.00313/) (2023)
- [On the Effect of Pruning on Adversarial Robustness](https://mlsanity.com/paper/2108.04890/) (2021)
- [Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness](https://mlsanity.com/paper/2203.07653/) (2022)
- [Enhance the Visual Representation via Discrete Adversarial Training](https://mlsanity.com/paper/2209.07735/) (2022)
- [Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning](https://mlsanity.com/paper/2308.02533/) (2023)
- [Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness](https://mlsanity.com/paper/2106.02734/) (2021)
- [SATO: Stable Text-to-Motion Framework](https://mlsanity.com/paper/2405.01461/) (2024)
- [Self-training Converts Weak Learners to Strong Learners in Mixture Models](https://mlsanity.com/paper/2106.13805/) (2021)
- [How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis](https://mlsanity.com/paper/2201.08514/) (2022)
- [Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothing](https://mlsanity.com/paper/2301.12554/) (2023)
- [Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification](https://mlsanity.com/paper/2205.09619/) (2022)
- [Instruct2Attack: Language-Guided Semantic Adversarial Attacks](https://mlsanity.com/paper/2311.15551/) (2023)
- [FRAPPE: A Group Fairness Framework for Post-Processing Everything](https://mlsanity.com/paper/2312.02592/) (2023)
- [MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers](https://mlsanity.com/paper/2402.02263/) (2024)
- [Interpolation can hurt robust generalization even when there is no noise](https://mlsanity.com/paper/2108.02883/) (2021)

## Abstract (arXiv)

Adversarial training augments the training set with perturbations to improve the robust error (over worst-case perturbations), but it often leads to an increase in the standard error (on unperturbed test inputs). Previous explanations for this tradeoff rely on the assumption that no predictor in the hypothesis class has low standard and robust error. In this work, we precisely characterize the effect of augmentation on the standard error in linear regression when the optimal linear predictor has zero standard and robust error. In particular, we show that the standard error could increase even when the augmented perturbations have noiseless observations from the optimal linear predictor. We then prove that the recently proposed robust self-training (RST) estimator improves robust error without sacrificing standard error for noiseless linear regression. Empirically, for neural networks, we find that RST with different adversarial training methods improves both standard and robust error for random and adversarial rotations and adversarial ℓ_∞ perturbations in CIFAR-10.