# Fairness Evaluation with Item Response Theory

arXiv: https://arxiv.org/abs/2411.02414 · HTML: https://mlsanity.com/paper/2411.02414/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 54 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [The Variational Fair Autoencoder](https://mlsanity.com/paper/1511.00830/) (2015)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Fair Inference On Outcomes](https://mlsanity.com/paper/1705.10378/) (2017)
- [A causal framework for discovering and removing direct and indirect discrimination](https://mlsanity.com/paper/1611.07509/) (2016)
- [FairGT: A Fairness-aware Graph Transformer](https://mlsanity.com/paper/2404.17169/) (2024)
- [β3-IRT: A New Item Response Model and its Applications](https://mlsanity.com/paper/1903.04016/) (2019)
- [Causal Inference with Conditional Front-Door Adjustment and Identifiable Variational Autoencoder](https://mlsanity.com/paper/2310.01937/) (2023)
- [How to be fair? A study of label and selection bias](https://mlsanity.com/paper/2403.14282/) (2024)
- [FUGNN: Harmonizing Fairness and Utility in Graph Neural Networks](https://mlsanity.com/paper/2405.17034/) (2024)
- [Assessing Classifier Fairness with Collider Bias](https://mlsanity.com/paper/2010.03933/) (2020)
- [Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference](https://mlsanity.com/paper/2302.11944/) (2023)
- [Disentangled Representation with Causal Constraints for Counterfactual Fairness](https://mlsanity.com/paper/2208.09147/) (2022)
- [Comprehensive Algorithm Portfolio Evaluation using Item Response Theory](https://mlsanity.com/paper/2307.15850/) (2023)

## Built on by (most cited first, 7 of 7)

- [Revisiting Pre-processing Group Fairness: A Modular Benchmarking Framework](https://mlsanity.com/paper/2508.15193/) (2025)
- [FairGE: Fairness-Aware Graph Encoding in Incomplete Social Networks](https://mlsanity.com/paper/2601.09394/) (2026)
- [Fairness in Augmented Graph Learning: A Survey](https://mlsanity.com/paper/2504.21296/) (2025)
- [When to Invoke: Refining LLM Fairness with Toxicity Assessment](https://mlsanity.com/paper/2601.09250/) (2026)
- [LiteFat: Lightweight Spatio-Temporal Graph Learning for Real-Time Driver Fatigue Detection](https://mlsanity.com/paper/2507.21756/) (2025)
- [Harnessing LLM for Noise-Robust Cognitive Diagnosis in Web-Based Intelligent Education Systems](https://mlsanity.com/paper/2510.04093/) (2025)
- [FairGC: Fairness-aware Graph Condensation](https://mlsanity.com/paper/2603.28321/) (2026)

## Abstract (arXiv)

Item Response Theory (IRT) has been widely used in educational psychometrics to assess student ability, as well as the difficulty and discrimination of test questions. In this context, discrimination specifically refers to how effectively a question distinguishes between students of different ability levels, and it does not carry any connotation related to fairness. In recent years, IRT has been successfully used to evaluate the predictive performance of Machine Learning (ML) models, but this paper marks its first application in fairness evaluation. In this paper, we propose a novel Fair-IRT framework to evaluate a set of predictive models on a set of individuals, while simultaneously eliciting specific parameters, namely, the ability to make fair predictions (a feature of predictive models), as well as the discrimination and difficulty of individuals that affect the prediction results. Furthermore, we conduct a series of experiments to comprehensively understand the implications of these parameters for fairness evaluation. Detailed explanations for item characteristic curves (ICCs) are provided for particular individuals. We propose the flatness of ICCs to disentangle the unfairness between individuals and predictive models. The experiments demonstrate the effectiveness of this framework as a fairness evaluation tool. Two real-world case studies illustrate its potential application in evaluating fairness in both classification and regression tasks. Our paper aligns well with the Responsible Web track by proposing a Fair-IRT framework to evaluate fairness in ML models, which directly contributes to the development of a more inclusive, equitable, and trustworthy AI.