# Comprehensive Algorithm Portfolio Evaluation using Item Response Theory

arXiv: https://arxiv.org/abs/2307.15850 · HTML: https://mlsanity.com/paper/2307.15850/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 52 references that are in the map.
- First posted on arXiv 29 Jul 2023.

## Builds on (most cited first, 6 of 6)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [ASlib: A Benchmark Library for Algorithm Selection](https://mlsanity.com/paper/1506.02465/) (2015)
- [β3-IRT: A New Item Response Model and its Applications](https://mlsanity.com/paper/1903.04016/) (2019)
- [OpenML: An R Package to Connect to the Machine Learning Platform OpenML](https://mlsanity.com/paper/1701.01293/) (2017)
- [Capturing Humans' Mental Models of AI: An Item Response Theory Approach](https://mlsanity.com/paper/2305.09064/) (2023)
- [Unsupervised Anomaly Detection Ensembles using Item Response Theory](https://mlsanity.com/paper/2106.06243/) (2021)

## Built on by (most cited first, 2 of 2)

- [Fairness Evaluation with Item Response Theory](https://mlsanity.com/paper/2411.02414/) (2024)
- [An Item Response Theory-based R Module for Algorithm Portfolio Analysis](https://mlsanity.com/paper/2408.14025/) (2024)

## Abstract (arXiv)

Item Response Theory (IRT) has been proposed within the field of Educational Psychometrics to assess student ability as well as test question difficulty and discrimination power. More recently, IRT has been applied to evaluate machine learning algorithm performance on a single classification dataset, where the student is now an algorithm, and the test question is an observation to be classified by the algorithm. In this paper we present a modified IRT-based framework for evaluating a portfolio of algorithms across a repository of datasets, while simultaneously eliciting a richer suite of characteristics - such as algorithm consistency and anomalousness - that describe important aspects of algorithm performance. These characteristics arise from a novel inversion and reinterpretation of the traditional IRT model without requiring additional dataset feature computations. We test this framework on algorithm portfolios for a wide range of applications, demonstrating the broad applicability of this method as an insightful algorithm evaluation tool. Furthermore, the explainable nature of IRT parameters yield an increased understanding of algorithm portfolios.