# On Evaluation of Vision Datasets and Models using Human Competency Frameworks

arXiv: https://arxiv.org/abs/2409.04041 · HTML: https://mlsanity.com/paper/2409.04041/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Builds on (most cited first, 9 of 9)

- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [Dynamic Data Selection for Curriculum Learning via Ability Estimation](https://mlsanity.com/paper/2011.00080/) (2020)
- [β3-IRT: A New Item Response Model and its Applications](https://mlsanity.com/paper/1903.04016/) (2019)
- [Item Response Theory based Ensemble in Machine Learning](https://mlsanity.com/paper/1911.04616/) (2019)
- [Unsupervised Anomaly Detection Ensembles using Item Response Theory](https://mlsanity.com/paper/2106.06243/) (2021)

## Abstract (arXiv)

Evaluating models and datasets in computer vision remains a challenging task, with most leaderboards relying solely on accuracy. While accuracy is a popular metric for model evaluation, it provides only a coarse assessment by considering a single model's score on all dataset items. This paper explores Item Response Theory (IRT), a framework that infers interpretable latent parameters for an ensemble of models and each dataset item, enabling richer evaluation and analysis beyond the single accuracy number. Leveraging IRT, we assess model calibration, select informative data subsets, and demonstrate the usefulness of its latent parameters for analyzing and comparing models and datasets in computer vision.