# Adversarial Weight Perturbation Helps Robust Generalization

arXiv: https://arxiv.org/abs/2004.05884 · HTML: https://mlsanity.com/paper/2004.05884/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 10 of 10)

- [Sparsity Winning Twice: Better Robust Generalization from More Efficient Training](https://mlsanity.com/paper/2202.09844/) (2022)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [Randomized Adversarial Training via Taylor Expansion](https://mlsanity.com/paper/2303.10653/) (2023)
- [Network insensitivity to parameter noise via adversarial regularization](https://mlsanity.com/paper/2106.05009/) (2021)
- [Data-Efficient Double-Win Lottery Tickets from Robust Pre-training](https://mlsanity.com/paper/2206.04762/) (2022)
- [Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search](https://mlsanity.com/paper/2304.05405/) (2023)
- [Enhancing Adversarial Training via Reweighting Optimization Trajectory](https://mlsanity.com/paper/2306.14275/) (2023)
- [WeiPer: OOD Detection using Weight Perturbations of Class Projections](https://mlsanity.com/paper/2405.17164/) (2024)
- [On robust overfitting: adversarial training induced distribution matters](https://mlsanity.com/paper/2311.16526/) (2023)
- [Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization](https://mlsanity.com/paper/2406.05372/) (2024)

## Abstract (arXiv)

The study on improving the robustness of deep neural networks against adversarial examples grows rapidly in recent years. Among them, adversarial training is the most promising one, which flattens the input loss landscape (loss change with respect to input) via training on adversarially perturbed examples. However, how the widely used weight loss landscape (loss change with respect to weight) performs in adversarial training is rarely explored. In this paper, we investigate the weight loss landscape from a new perspective, and identify a clear correlation between the flatness of weight loss landscape and robust generalization gap. Several well-recognized adversarial training improvements, such as early stopping, designing new objective functions, or leveraging unlabeled data, all implicitly flatten the weight loss landscape. Based on these observations, we propose a simple yet effective Adversarial Weight Perturbation (AWP) to explicitly regularize the flatness of weight loss landscape, forming a double-perturbation mechanism in the adversarial training framework that adversarially perturbs both inputs and weights. Extensive experiments demonstrate that AWP indeed brings flatter weight loss landscape and can be easily incorporated into various existing adversarial training methods to further boost their adversarial robustness.