# Are All Training Examples Created Equal? An Empirical Study

arXiv: https://arxiv.org/abs/1811.12569 · HTML: https://mlsanity.com/paper/1811.12569/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2018.

## Built on by (most cited first, 16 of 16)

- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data](https://mlsanity.com/paper/2103.03399/) (2021)
- [Towards a statistical theory of data selection under weak supervision](https://mlsanity.com/paper/2309.14563/) (2023)
- [Algorithm Fairness in AI for Medicine and Healthcare](https://mlsanity.com/paper/2110.00603/) (2021)
- [Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuning](https://mlsanity.com/paper/2407.06120/) (2024)
- [One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning](https://mlsanity.com/paper/2104.13114/) (2021)
- [Improving Data Quality with Training Dynamics of Gradient Boosting Decision Trees](https://mlsanity.com/paper/2210.11327/) (2022)
- [Reduce, Reuse, Recycle: Improving Training Efficiency with Distillation](https://mlsanity.com/paper/2211.00683/) (2022)
- [Exploring Data Redundancy in Real-world Image Classification through Data Selection](https://mlsanity.com/paper/2306.14113/) (2023)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [How Low Can We Go? Pixel Annotation for Semantic Segmentation](https://mlsanity.com/paper/2201.10448/) (2022)
- [Semantically Redundant Training Data Removal and Deep Model Classification Performance: A Study with Chest X-rays](https://mlsanity.com/paper/2309.09773/) (2023)
- [On the Importance of Large Objects in CNN Based Object Detection Algorithms](https://mlsanity.com/paper/2311.11714/) (2023)
- [Automatic Pruning of Fine-tuning Datasets for Transformer-based Language Models](https://mlsanity.com/paper/2407.08887/) (2024)
- [textitGeoHard: Towards Measuring Class-wise Hardness through Modelling Class Semantics](https://mlsanity.com/paper/2407.12512/) (2024)
- [AGI-Elo: How Far Are We From Mastering A Task?](https://mlsanity.com/paper/2505.12844/) (2025)

## Abstract (arXiv)

Modern computer vision algorithms often rely on very large training datasets. However, it is conceivable that a carefully selected subsample of the dataset is sufficient for training. In this paper, we propose a gradient-based importance measure that we use to empirically analyze relative importance of training images in four datasets of varying complexity. We find that in some cases, a small subsample is indeed sufficient for training. For other datasets, however, the relative differences in importance are negligible. These results have important implications for active learning on deep networks. Additionally, our analysis method can be used as a general tool to better understand diversity of training examples in datasets.