# A Gradient-based Approach for Online Robust Deep Neural Network Training with Noisy Labels

arXiv: https://arxiv.org/abs/2306.05046 · HTML: https://mlsanity.com/paper/2306.05046/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 23 of 23)

- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Iterative Learning with Open-set Noisy Labels](https://mlsanity.com/paper/1804.00092/) (2018)
- [Online Deep Clustering for Unsupervised Representation Learning](https://mlsanity.com/paper/2006.10645/) (2020)
- [Webly Supervised Learning of Convolutional Networks](https://mlsanity.com/paper/1505.01554/) (2015)
- [Online Convex Optimization with Stochastic Constraints](https://mlsanity.com/paper/1708.03741/) (2017)
- [Online Deep Learning: Learning Deep Neural Networks on the Fly](https://mlsanity.com/paper/1711.03705/) (2017)
- [A Topological Filter for Learning with Label Noise](https://mlsanity.com/paper/2012.04835/) (2020)
- [Efficient Regret Minimization in Non-Convex Games](https://mlsanity.com/paper/1708.00075/) (2017)
- [Towards Robust Neural Networks via Close-loop Control](https://mlsanity.com/paper/2102.01862/) (2021)
- [Data-Driven Distributionally Robust Electric Vehicle Balancing for Mobility-on-Demand Systems under Demand and Supply Uncertainties](https://mlsanity.com/paper/2210.10887/) (2022)
- [Particle-based Online Bayesian Sampling](https://mlsanity.com/paper/2302.14796/) (2023)
- [The Dark Side of Explanations: Poisoning Recommender Systems with Counterfactual Examples](https://mlsanity.com/paper/2305.00574/) (2023)
- [Regrets of Proximal Method of Multipliers for Online Non-convex Optimization with Long Term Constraints](https://mlsanity.com/paper/2204.10986/) (2022)
- [Data-Driven Distributionally Robust Electric Vehicle Balancing for Autonomous Mobility-on-Demand Systems under Demand and Supply Uncertainties](https://mlsanity.com/paper/2211.13797/) (2022)
- [Contrastive Learning with Consistent Representations](https://mlsanity.com/paper/2302.01541/) (2023)
- [MR-Based Electrical Property Reconstruction Using Physics-Informed Neural Networks](https://mlsanity.com/paper/2210.12584/) (2022)
- [Tensorized Optical Multimodal Fusion Network](https://mlsanity.com/paper/2302.08744/) (2023)

## Built on by (most cited first, 1 of 1)

- [CAdam: Confidence-Based Optimization for Online Learning](https://mlsanity.com/paper/2411.19647/) (2024)

## Abstract (arXiv)

Learning with noisy labels is an important topic for scalable training in many real-world scenarios. However, few previous research considers this problem in the online setting, where the arrival of data is streaming. In this paper, we propose a novel gradient-based approach to enable the detection of noisy labels for the online learning of model parameters, named Online Gradient-based Robust Selection (OGRS). In contrast to the previous sample selection approach for the offline training that requires the estimation of a clean ratio of the dataset before each epoch of training, OGRS can automatically select clean samples by steps of gradient update from datasets with varying clean ratios without changing the parameter setting. During the training process, the OGRS method selects clean samples at each iteration and feeds the selected sample to incrementally update the model parameters. We provide a detailed theoretical analysis to demonstrate data selection process is converging to the low-loss region of the sample space, by introducing and proving the sub-linear local Lagrangian regret of the non-convex constrained optimization problem. Experimental results show that it outperforms state-of-the-art methods in different settings.