# Was that so hard? Estimating human classification difficulty

arXiv: https://arxiv.org/abs/2203.11824 · HTML: https://mlsanity.com/paper/2203.11824/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [PyTorch Metric Learning](https://mlsanity.com/paper/2008.09164/) (2020)
- [How hard can it be? Estimating the difficulty of visual search in an image](https://mlsanity.com/paper/1705.08280/) (2017)
- [Divide and Conquer the Embedding Space for Metric Learning](https://mlsanity.com/paper/1906.05990/) (2019)
- [A survey of active learning algorithms for supervised remote sensing image classification](https://mlsanity.com/paper/2104.07784/) (2021)
- [Efficient Image Dataset Classification Difficulty Estimation for Predicting Deep-Learning Accuracy](https://mlsanity.com/paper/1803.09588/) (2018)
- [Statistical Measures For Defining Curriculum Scoring Function](https://mlsanity.com/paper/2103.00147/) (2021)

## Built on by (most cited first, 3 of 3)

- [A Lightweight Measure of Classification Difficulty from Application Dataset Characteristics](https://mlsanity.com/paper/2404.05981/) (2024)
- [Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation](https://mlsanity.com/paper/2506.20449/) (2025)
- [Efficient Neural Network Model Selection for Few-Class Application Datasets](https://mlsanity.com/paper/2606.19712/) (2026)

## Abstract (arXiv)

When doctors are trained to diagnose a specific disease, they learn faster when presented with cases in order of increasing difficulty. This creates the need for automatically estimating how difficult it is for doctors to classify a given case. In this paper, we introduce methods for estimating how hard it is for a doctor to diagnose a case represented by a medical image, both when ground truth difficulties are available for training, and when they are not. Our methods are based on embeddings obtained with deep metric learning. Additionally, we introduce a practical method for obtaining ground truth human difficulty for each image case in a dataset using self-assessed certainty. We apply our methods to two different medical datasets, achieving high Kendall rank correlation coefficients, showing that we outperform existing methods by a large margin on our problem and data.