# Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee

arXiv: https://arxiv.org/abs/1905.11368 · HTML: https://mlsanity.com/paper/1905.11368/

## Key facts

- Cited by 3 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 3 of 3)

- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network](https://mlsanity.com/paper/2312.03386/) (2023)
- [Weighted Neural Tangent Kernel: A Generalized and Improved Network-Induced Kernel](https://mlsanity.com/paper/2103.11558/) (2021)

## Abstract (arXiv)

Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. Such over-fitting ability hinders generalization when mislabeled training examples are present. On the other hand, simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios, a phenomenon not theoretically understood. This paper proposes and analyzes two simple and intuitive regularization methods: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels. Our generalization analysis relies on the connection between wide neural network and neural tangent kernel (NTK). The generalization bound is independent of the network size, and is comparable to the bound one can get when there is no label noise. Experimental results verify the effectiveness of these methods on noisily labeled datasets.