# Coresets for Robust Training of Neural Networks against Noisy Labels

arXiv: https://arxiv.org/abs/2011.07451 · HTML: https://mlsanity.com/paper/2011.07451/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2020.

## Built on by (most cited first, 24 of 24)

- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [FINE Samples for Learning with Noisy Labels](https://mlsanity.com/paper/2102.11628/) (2021)
- [Sample Selection with Uncertainty of Losses for Learning with Noisy Labels](https://mlsanity.com/paper/2106.00445/) (2021)
- [Learning Fast Sample Re-weighting Without Reward Data](https://mlsanity.com/paper/2109.03216/) (2021)
- [SELC: Self-Ensemble Label Correction Improves Learning with Noisy Labels](https://mlsanity.com/paper/2205.01156/) (2022)
- [Identifying Hard Noise in Long-Tailed Sample Distribution](https://mlsanity.com/paper/2207.13378/) (2022)
- [Tackling Noisy Labels with Network Parameter Additive Decomposition](https://mlsanity.com/paper/2403.13241/) (2024)
- [Obstacle Aware Sampling for Path Planning](https://mlsanity.com/paper/2203.04075/) (2022)
- [GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning](https://mlsanity.com/paper/2301.13465/) (2023)
- [Minimax Optimization: The Case of Convex-Submodular](https://mlsanity.com/paper/2111.01262/) (2021)
- [Learning to Combat Noisy Labels via Classification Margins](https://mlsanity.com/paper/2102.00751/) (2021)
- [A Gift from Label Smoothing: Robust Training with Adaptive Label Smoothing via Auxiliary Classifier under Label Noise](https://mlsanity.com/paper/2206.07277/) (2022)
- [Sample Selection via Contrastive Fragmentation for Noisy Label Regression](https://mlsanity.com/paper/2502.17771/) (2025)
- [Learning to Selectively Learn for Weakly-supervised Paraphrase Generation](https://mlsanity.com/paper/2109.12457/) (2021)
- [MILD: Modeling the Instance Learning Dynamics for Learning with Noisy Labels](https://mlsanity.com/paper/2306.11560/) (2023)
- [Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled yet Hard-to-Learn Samples in Noisy Data](https://mlsanity.com/paper/2504.17474/) (2025)
- [Soft Random Sampling: A Theoretical and Empirical Analysis](https://mlsanity.com/paper/2311.12727/) (2023)
- [Denoising after Entropy-based Debiasing A Robust Training Method for Dataset Bias with Noisy Labels](https://mlsanity.com/paper/2212.01189/) (2022)
- [Density-Aware Farthest Point Sampling](https://mlsanity.com/paper/2509.13213/) (2025)
- [Efficient NTK using Dimensionality Reduction](https://mlsanity.com/paper/2210.04807/) (2022)
- [Mitigating Noisy Supervision Using Synthetic Samples with Soft Labels](https://mlsanity.com/paper/2406.16966/) (2024)
- [Deterministic Coreset for Lp Subspace](https://mlsanity.com/paper/2601.00361/) (2026)
- [Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuning](https://mlsanity.com/paper/2610.00436/) (2026)

## Abstract (arXiv)

Modern neural networks have the capacity to overfit noisy labels frequently found in real-world datasets. Although great progress has been made, existing techniques are limited in providing theoretical guarantees for the performance of the neural networks trained with noisy labels. Here we propose a novel approach with strong theoretical guarantees for robust training of deep networks trained with noisy labels. The key idea behind our method is to select weighted subsets (coresets) of clean data points that provide an approximately low-rank Jacobian matrix. We then prove that gradient descent applied to the subsets do not overfit the noisy labels. Our extensive experiments corroborate our theory and demonstrate that deep networks trained on our subsets achieve a significantly superior performance compared to state-of-the art, e.g., 6% increase in accuracy on CIFAR-10 with 80% noisy labels, and 7% increase in accuracy on mini Webvision.