# Which Samples Should be Learned First: Easy or Hard?

arXiv: https://arxiv.org/abs/2110.05481 · HTML: https://mlsanity.com/paper/2110.05481/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 100 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://mlsanity.com/paper/1907.10903/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling](https://mlsanity.com/paper/1801.10247/) (2018)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View](https://mlsanity.com/paper/1909.03211/) (2019)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Not All Samples Are Created Equal: Deep Learning with Importance Sampling](https://mlsanity.com/paper/1803.00942/) (2018)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)

## Built on by (most cited first, 12 of 12)

- [On Diversified Preferences of Large Language Model Alignment](https://mlsanity.com/paper/2312.07401/) (2023)
- [Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure](https://mlsanity.com/paper/2301.04850/) (2023)
- [Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective](https://mlsanity.com/paper/2401.04374/) (2024)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [Enhancing In-Context Learning via Implicit Demonstration Augmentation](https://mlsanity.com/paper/2407.00100/) (2024)
- [General Cyclical Training of Neural Networks](https://mlsanity.com/paper/2202.08835/) (2022)
- [Exploring the Learning Difficulty of Data Theory and Measure](https://mlsanity.com/paper/2205.07427/) (2022)
- [Dynamic Prototype Rehearsal for Continual ECG Arrhythmia Detection](https://mlsanity.com/paper/2501.07555/) (2025)
- [A Safety-Adapted Loss for Pedestrian Detection in Automated Driving](https://mlsanity.com/paper/2402.02986/) (2024)
- [When Active Learning Fails, Uncalibrated Out of Distribution Uncertainty Quantification Might Be the Problem](https://mlsanity.com/paper/2511.17760/) (2025)
- [LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training](https://mlsanity.com/paper/2509.20786/) (2025)
- [Evolution-Inspired Sample Competition for Deep Neural Network Optimization](https://mlsanity.com/paper/2604.12568/) (2026)

## Abstract (arXiv)

An effective weighting scheme for training samples is essential for learning tasks. Numerous weighting schemes have been proposed. Some schemes take the easy-first mode, whereas some others take the hard-first one. Naturally, an interesting yet realistic question is raised. Which samples should be learned first given a new learning task, easy or hard? To answer this question, both theoretical analyses and experimental verification are conducted. First, a general optimized objective function is proposed, revealing the relationship between the difficulty distribution and the difficulty-based sample weights. Second, on the basis of the optimized objective function, theoretical answers are obtained. Besides the easy-first and hard-first modes, there are two other priority modes, namely, medium-first and two-ends-first. The prior mode does not necessarily remain unchanged during the training process. Third, an effective and universal solution is proposed to select the optimal priority mode when there is no prior knowledge or theoretical clues. The four modes, namely, easy/medium/hard/two-ends-first, can be flexibly switched in the proposed solution. Fourth, a wide range of experiments is conducted under various scenarios to further compare the weighting schemes in different modes. On the basis of these works, reasonable and comprehensive answers are obtained. Factors including the distribution of samples' learning difficulties and the validation data determine which samples should be learned first in a learning task.