# Inductive Bias of Gradient Descent based Adversarial Training on Separable Data

arXiv: https://arxiv.org/abs/1906.02931 · HTML: https://mlsanity.com/paper/1906.02931/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2019.

## Built on by (most cited first, 12 of 12)

- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Improved OOD Generalization via Adversarial Training and Pre-training](https://mlsanity.com/paper/2105.11144/) (2021)
- [Adversarial robustness of sparse local Lipschitz predictors](https://mlsanity.com/paper/2202.13216/) (2022)
- [Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition](https://mlsanity.com/paper/2103.09947/) (2021)
- [Provable Robustness of Adversarial Training for Learning Halfspaces with Noise](https://mlsanity.com/paper/2104.09437/) (2021)
- [Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach](https://mlsanity.com/paper/2104.04886/) (2021)
- [Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Data](https://mlsanity.com/paper/2108.06808/) (2021)
- [Learning to be adversarially robust and differentially private](https://mlsanity.com/paper/2201.02265/) (2022)
- [On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamics](https://mlsanity.com/paper/2306.01271/) (2023)
- [ARCH: Efficient Adversarial Regularized Training with Caching](https://mlsanity.com/paper/2109.07048/) (2021)
- [On robust overfitting: adversarial training induced distribution matters](https://mlsanity.com/paper/2311.16526/) (2023)
- [Robustness Cannot be Reduced to Regularization: Studying Adversarial Training Beyond the Linear Case](https://mlsanity.com/paper/2606.21488/) (2026)

## Abstract (arXiv)

Adversarial training is a principled approach for training robust neural networks. Despite of tremendous successes in practice, its theoretical properties still remain largely unexplored. In this paper, we provide new theoretical insights of gradient descent based adversarial training by studying its computational properties, specifically on its inductive bias. We take the binary classification task on linearly separable data as an illustrative example, where the loss asymptotically attains its infimum as the parameter diverges to infinity along certain directions. Specifically, we show that when the adversarial perturbation during training has bounded ℓ2-norm, the classifier learned by gradient descent based adversarial training converges in direction to the maximum ℓ2-norm margin classifier at the rate of tildeO(1/√T), significantly faster than the rate O(1/log T) of training with clean data. In addition, when the adversarial perturbation during training has bounded ℓq-norm for some q≥ 1, the resulting classifier converges in direction to a maximum mixed-norm margin classifier, which has a natural interpretation of robustness, as being the maximum ℓ2-norm margin classifier under worst-case ℓq-norm perturbation to the data. Our findings provide theoretical backups for adversarial training that it indeed promotes robustness against adversarial perturbation.