# Self-Adaptive Training: beyond Empirical Risk Minimization

arXiv: https://arxiv.org/abs/2002.10319 · HTML: https://mlsanity.com/paper/2002.10319/

## Key facts

- Cited by 126 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 30 of 126)

- [Fixing Data Augmentation to Improve Adversarial Robustness](https://mlsanity.com/paper/2103.01946/) (2021)
- [Data Augmentation Can Improve Robustness](https://mlsanity.com/paper/2111.05328/) (2021)
- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [Robust Training under Label Noise by Over-parameterization](https://mlsanity.com/paper/2202.14026/) (2022)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses](https://mlsanity.com/paper/2305.10862/) (2023)
- [Exploring Memorization in Adversarial Training](https://mlsanity.com/paper/2106.01606/) (2021)
- [Improved Regularization and Robustness for Fine-tuning in Neural Networks](https://mlsanity.com/paper/2111.04578/) (2021)
- [Practical Evaluation of Adversarial Robustness via Adaptive Auto Attack](https://mlsanity.com/paper/2203.05154/) (2022)
- [A Unified Wasserstein Distributional Robustness Framework for Adversarial Training](https://mlsanity.com/paper/2202.13437/) (2022)
- [Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees](https://mlsanity.com/paper/2206.02659/) (2022)
- [DISCO: Adversarial Defense with Local Implicit Functions](https://mlsanity.com/paper/2212.05630/) (2022)
- [Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attacks](https://mlsanity.com/paper/2105.10304/) (2021)
- [Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance](https://mlsanity.com/paper/2208.08664/) (2022)
- [Attacking Adversarial Attacks as A Defense](https://mlsanity.com/paper/2106.04938/) (2021)
- [Towards Better Selective Classification](https://mlsanity.com/paper/2206.09034/) (2022)
- [Robust Models are less Over-Confident](https://mlsanity.com/paper/2210.05938/) (2022)
- [LAFEAT: Piercing Through Adversarial Defenses with Latent Features](https://mlsanity.com/paper/2104.09284/) (2021)
- [CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning](https://mlsanity.com/paper/2202.05613/) (2022)
- [Neighborhood Collective Estimation for Noisy Label Identification and Correction](https://mlsanity.com/paper/2208.03207/) (2022)
- [Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning](https://mlsanity.com/paper/2312.09505/) (2023)
- [Better Supervisory Signals by Observing Learning Paths](https://mlsanity.com/paper/2203.02485/) (2022)
- [Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection](https://mlsanity.com/paper/2111.14932/) (2021)
- [CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters](https://mlsanity.com/paper/2203.15331/) (2022)
- [Selective Prediction via Training Dynamics](https://mlsanity.com/paper/2205.13532/) (2022)
- [Understanding the Interaction of Adversarial Training with Noisy Labels](https://mlsanity.com/paper/2102.03482/) (2021)
- [Adversarial Robustness via Fisher-Rao Regularization](https://mlsanity.com/paper/2106.06685/) (2021)
- [Two Wrongs Don't Make a Right: Combating Confirmation Bias in Learning with Label Noise](https://mlsanity.com/paper/2112.02960/) (2021)
- [Revisiting adapters with adversarial training](https://mlsanity.com/paper/2210.04886/) (2022)
- [Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning](https://mlsanity.com/paper/2101.08732/) (2021)

## Abstract (arXiv)

We propose self-adaptive training---a new training algorithm that dynamically corrects problematic training labels by model predictions without incurring extra computational cost---to improve generalization of deep learning for potentially corrupted training data. This problem is crucial towards robustly learning from data that are corrupted by, e.g., label noises and out-of-distribution samples. The standard empirical risk minimization (ERM) for such data, however, may easily overfit noises and thus suffers from sub-optimal performance. In this paper, we observe that model predictions can substantially benefit the training process: self-adaptive training significantly improves generalization over ERM under various levels of noises, and mitigates the overfitting issue in both natural and adversarial training. We evaluate the error-capacity curve of self-adaptive training: the test error is monotonously decreasing w.r.t. model capacity. This is in sharp contrast to the recently-discovered double-descent phenomenon in ERM which might be a result of overfitting of noises. Experiments on CIFAR and ImageNet datasets verify the effectiveness of our approach in two applications: classification with label noise and selective classification. We release our code at https://github.com/LayneH/self-adaptive-training.