# Empirical Study of Easy and Hard Examples in CNN Training

arXiv: https://arxiv.org/abs/1911.10739 · HTML: https://mlsanity.com/paper/1911.10739/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2019.

## Built on by (most cited first, 9 of 9)

- [Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyond](https://mlsanity.com/paper/2503.07501/) (2025)
- [What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization](https://mlsanity.com/paper/2207.05161/) (2022)
- [What Images are More Memorable to Machines?](https://mlsanity.com/paper/2211.07625/) (2022)
- [Differences Between Hard and Noisy-labeled Samples: An Empirical Study](https://mlsanity.com/paper/2307.10718/) (2023)
- [SoK: Memorisation in machine learning](https://mlsanity.com/paper/2311.03075/) (2023)
- [Angel or Devil: Discriminating Hard Samples and Anomaly Contaminations for Unsupervised Time Series Anomaly Detection](https://mlsanity.com/paper/2410.21322/) (2024)
- [SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuning](https://mlsanity.com/paper/2603.18423/) (2026)
- [Unified Regularity Measures for Sample-wise Learning and Generalization](https://mlsanity.com/paper/2108.03913/) (2021)
- [Easy Batch Normalization](https://mlsanity.com/paper/2207.08940/) (2022)

## Abstract (arXiv)

Deep Neural Networks (DNNs) generalize well despite their massive size and capability of memorizing all examples. There is a hypothesis that DNNs start learning from simple patterns and the hypothesis is based on the existence of examples that are consistently well-classified at the early training stage (i.e., easy examples) and examples misclassified (i.e., hard examples). Easy examples are the evidence that DNNs start learning from specific patterns and there is a consistent learning process. It is important to know how DNNs learn patterns and obtain generalization ability, however, properties of easy and hard examples are not thoroughly investigated (e.g., contributions to generalization and visual appearances). In this work, we study the similarities of easy and hard examples respectively for different Convolutional Neural Network (CNN) architectures, assessing how those examples contribute to generalization. Our results show that easy examples are visually similar to each other and hard examples are visually diverse, and both examples are largely shared across different CNN architectures. Moreover, while hard examples tend to contribute more to generalization than easy examples, removing a large number of easy examples leads to poor generalization. By analyzing those results, we hypothesize that biases in a dataset and Stochastic Gradient Descent (SGD) are the reasons why CNNs have consistent easy and hard examples. Furthermore, we show that large scale classification datasets can be efficiently compressed by using easiness proposed in this work.