# Exploring the Learning Difficulty of Data Theory and Measure

arXiv: https://arxiv.org/abs/2205.07427 · HTML: https://mlsanity.com/paper/2205.07427/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 83 references that are in the map.
- First posted on arXiv 16 May 2022.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Geometry-aware Instance-reweighted Adversarial Training](https://mlsanity.com/paper/2010.01736/) (2020)
- [Deep learning generalizes because the parameter-function map is biased towards simple functions](https://mlsanity.com/paper/1805.08522/) (2018)
- [A Survey of Label-noise Representation Learning: Past, Present and Future](https://mlsanity.com/paper/2011.04406/) (2020)
- [Learning with Symmetric Label Noise: The Importance of Being Unhinged](https://mlsanity.com/paper/1505.07634/) (2015)
- [Gradient Harmonized Single-stage Detector](https://mlsanity.com/paper/1811.05181/) (2018)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [Rethinking Bias-Variance Trade-off for Generalization of Neural Networks](https://mlsanity.com/paper/2002.11328/) (2020)
- [Estimating Example Difficulty Using Variance of Gradients](https://mlsanity.com/paper/2008.11600/) (2020)
- [Dynamic Curriculum Learning for Imbalanced Data Classification](https://mlsanity.com/paper/1901.06783/) (2019)
- [Denoising Implicit Feedback for Recommendation](https://mlsanity.com/paper/2006.04153/) (2020)
- [Quantifying Uncertainties in Natural Language Processing Tasks](https://mlsanity.com/paper/1811.07253/) (2018)
- [Ranked List Loss for Deep Metric Learning](https://mlsanity.com/paper/1903.03238/) (2019)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping](https://mlsanity.com/paper/2102.11343/) (2021)
- [NSCaching: Simple and Efficient Negative Sampling for Knowledge Graph Embedding](https://mlsanity.com/paper/1812.06410/) (2018)
- [Understanding the role of importance weighting for deep learning](https://mlsanity.com/paper/2103.15209/) (2021)
- [Self-paced Ensemble for Highly Imbalanced Massive Data Classification](https://mlsanity.com/paper/1909.03500/) (2019)
- [On the Generalization Mystery in Deep Learning](https://mlsanity.com/paper/2203.10036/) (2022)
- [A Tale Of Two Long Tails](https://mlsanity.com/paper/2107.13098/) (2021)
- [A Unified Approach to Coreset Learning](https://mlsanity.com/paper/2111.03044/) (2021)
- [Unsupervised Domain Adaptive Salient Object Detection Through Uncertainty-Aware Pseudo-Label Learning](https://mlsanity.com/paper/2202.13170/) (2022)

## Built on by (most cited first, 4 of 4)

- [Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples](https://mlsanity.com/paper/2502.09650/) (2025)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [Reducing Class Bias In Data-Balanced Datasets Through Hardness-Based Resampling](https://mlsanity.com/paper/2504.07031/) (2025)
- [LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training](https://mlsanity.com/paper/2509.20786/) (2025)

## Abstract (arXiv)

As learning difficulty is crucial for machine learning (e.g., difficulty-based weighting learning strategies), previous literature has proposed a number of learning difficulty measures. However, no comprehensive investigation for learning difficulty is available to date, resulting in that nearly all existing measures are heuristically defined without a rigorous theoretical foundation. In addition, there is no formal definition of easy and hard samples even though they are crucial in many studies. This study attempts to conduct a pilot theoretical study for learning difficulty of samples. First, a theoretical definition of learning difficulty is proposed on the basis of the bias-variance trade-off theory on generalization error. Theoretical definitions of easy and hard samples are established on the basis of the proposed definition. A practical measure of learning difficulty is given as well inspired by the formal definition. Second, the properties for learning difficulty-based weighting strategies are explored. Subsequently, several classical weighting methods in machine learning can be well explained on account of explored properties. Third, the proposed measure is evaluated to verify its reasonability and superiority in terms of several main difficulty factors. The comparison in these experiments indicates that the proposed measure significantly outperforms the other measures throughout the experiments.