# Decoding machine learning benchmarks

arXiv: https://arxiv.org/abs/2007.14870 · HTML: https://mlsanity.com/paper/2007.14870/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2020.

## Built on by (most cited first, 8 of 8)

- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [Explanation-by-Example Based on Item Response Theory](https://mlsanity.com/paper/2210.01638/) (2022)
- [Explanations Based on Item Response Theory (eXirt): A Model-Specific Method to Explain Tree-Ensemble Model in Trust Perspective](https://mlsanity.com/paper/2210.09933/) (2022)
- [Data vs classifiers, who wins?](https://mlsanity.com/paper/2107.07451/) (2021)
- [How Reliable and Stable are Explanations of XAI Methods?](https://mlsanity.com/paper/2407.03108/) (2024)
- [Standing on the shoulders of giants](https://mlsanity.com/paper/2409.03151/) (2024)
- [Enhancing Classifier Evaluation: A Fairer Benchmarking Strategy Based on Ability and Robustness](https://mlsanity.com/paper/2504.09759/) (2025)
- [Beyond Random Sampling: Instance Quality-Based Data Partitioning via Item Response Theory](https://mlsanity.com/paper/2508.10628/) (2025)

## Abstract (arXiv)

Despite the availability of benchmark machine learning (ML) repositories (e.g., UCI, OpenML), there is no standard evaluation strategy yet capable of pointing out which is the best set of datasets to serve as gold standard to test different ML algorithms. In recent studies, Item Response Theory (IRT) has emerged as a new approach to elucidate what should be a good ML benchmark. This work applied IRT to explore the well-known OpenML-CC18 benchmark to identify how suitable it is on the evaluation of classifiers. Several classifiers ranging from classical to ensembles ones were evaluated using IRT models, which could simultaneously estimate dataset difficulty and classifiers' ability. The Glicko-2 rating system was applied on the top of IRT to summarize the innate ability and aptitude of classifiers. It was observed that not all datasets from OpenML-CC18 are really useful to evaluate classifiers. Most datasets evaluated in this work (84%) contain easy instances in general (e.g., around 10% of difficult instances only). Also, 80% of the instances in half of this benchmark are very discriminating ones, which can be of great use for pairwise algorithm comparison, but not useful to push classifiers abilities. This paper presents this new evaluation methodology based on IRT as well as the tool decodIRT, developed to guide IRT estimation over ML benchmarks.