# Differences Between Hard and Noisy-labeled Samples: An Empirical Study

arXiv: https://arxiv.org/abs/2307.10718 · HTML: https://mlsanity.com/paper/2307.10718/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 30 of 31)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Deep learning with noisy labels: exploring techniques and remedies in medical image analysis](https://mlsanity.com/paper/1912.02911/) (2019)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples](https://mlsanity.com/paper/1704.07433/) (2017)
- [Online Batch Selection for Faster Training of Neural Networks](https://mlsanity.com/paper/1511.06343/) (2015)
- [A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspective](https://mlsanity.com/paper/1811.03402/) (2018)
- [Combating Label Noise in Deep Learning Using Abstention](https://mlsanity.com/paper/1905.10964/) (2019)
- [Hard Sample Aware Noise Robust Learning for Histopathology Image Classification](https://mlsanity.com/paper/2112.03694/) (2021)
- [Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization](https://mlsanity.com/paper/2002.10657/) (2020)
- [Label Noise Types and Their Effects on Deep Learning](https://mlsanity.com/paper/2003.10471/) (2020)
- [A Survey on Deep Learning with Noisy Labels: How to train your model when you cannot trust on the annotations?](https://mlsanity.com/paper/2012.03061/) (2020)
- [SELC: Self-Ensemble Label Correction Improves Learning with Noisy Labels](https://mlsanity.com/paper/2205.01156/) (2022)
- [Salvage Reusable Samples from Noisy Data for Robust Learning](https://mlsanity.com/paper/2008.02427/) (2020)
- [Empirical Study of Easy and Hard Examples in CNN Training](https://mlsanity.com/paper/1911.10739/) (2019)

## Built on by (most cited first, 4 of 4)

- [Learning Discriminative Dynamics with Label Corruption for Noisy Label Detection](https://mlsanity.com/paper/2405.19902/) (2024)
- [Angel or Devil: Discriminating Hard Samples and Anomaly Contaminations for Unsupervised Time Series Anomaly Detection](https://mlsanity.com/paper/2410.21322/) (2024)
- [Discovering Learning-Friendly Generation Orders for Sequential Computation](https://mlsanity.com/paper/2506.23875/) (2025)
- [The Blind Spot in 2D Infants' Pose Estimation:Robust Learning from Noisy Annotations](https://mlsanity.com/paper/2609.04009/) (2026)

## Abstract (arXiv)

Extracting noisy or incorrectly labeled samples from a labeled dataset with hard/difficult samples is an important yet under-explored topic. Two general and often independent lines of work exist, one focuses on addressing noisy labels, and another deals with hard samples. However, when both types of data are present, most existing methods treat them equally, which results in a decline in the overall performance of the model. In this paper, we first design various synthetic datasets with custom hardness and noisiness levels for different samples. Our proposed systematic empirical study enables us to better understand the similarities and more importantly the differences between hard-to-learn samples and incorrectly-labeled samples. These controlled experiments pave the way for the development of methods that distinguish between hard and noisy samples. Through our study, we introduce a simple yet effective metric that filters out noisy-labeled samples while keeping the hard samples. We study various data partitioning methods in the presence of label noise and observe that filtering out noisy samples from hard samples with this proposed metric results in the best datasets as evidenced by the high test accuracy achieved after models are trained on the filtered datasets. We demonstrate this for both our created synthetic datasets and for datasets with real-world label noise. Furthermore, our proposed data partitioning method significantly outperforms other methods when employed within a semi-supervised learning framework.