# Using predictive multiplicity to measure individual performance within the AI Act

arXiv: https://arxiv.org/abs/2602.11944 · HTML: https://mlsanity.com/paper/2602.11944/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 70 references that are in the map.
- First posted on arXiv 12 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [Optimal Sparse Decision Trees](https://mlsanity.com/paper/1904.12847/) (2019)
- [On the Existence of Simpler Machine Learning Models](https://mlsanity.com/paper/1908.01755/) (2019)
- [Exploring the Whole Rashomon Set of Sparse Decision Trees](https://mlsanity.com/paper/2209.08040/) (2022)
- [Amazing Things Come From Having Many Good Models](https://mlsanity.com/paper/2407.04846/) (2024)
- [Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts](https://mlsanity.com/paper/2201.10295/) (2022)
- [Rashomon Capacity: A Metric for Predictive Multiplicity in Classification](https://mlsanity.com/paper/2206.01295/) (2022)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification](https://mlsanity.com/paper/2301.11562/) (2023)
- [Exploring and Interacting with the Set of Good Sparse Generalized Additive Models](https://mlsanity.com/paper/2303.16047/) (2023)
- [Arbitrary Decisions are a Hidden Cost of Differentially Private Training](https://mlsanity.com/paper/2302.14517/) (2023)
- [Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation](https://mlsanity.com/paper/2402.00728/) (2024)
- [The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions](https://mlsanity.com/paper/2304.10655/) (2023)
- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Robustness and Cybersecurity in the EU Artificial Intelligence Act](https://mlsanity.com/paper/2502.16184/) (2025)
- [What Constitutes a Less Discriminatory Algorithm?](https://mlsanity.com/paper/2412.18138/) (2024)
- [Non-Determinism and the Lawlessness of Machine Learning Code](https://mlsanity.com/paper/2206.11834/) (2022)
- [The Legal Duty to Search for Less Discriminatory Algorithms](https://mlsanity.com/paper/2406.06817/) (2024)
- [Observational Multiplicity](https://mlsanity.com/paper/2507.23136/) (2025)

## Built on by (most cited first, 2 of 2)

- [Mitigating the Multiplicity Burden: The Role of Calibration in Reducing Predictive Multiplicity of Classifiers](https://mlsanity.com/paper/2603.11750/) (2026)
- [Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packages](https://mlsanity.com/paper/2609.36970/) (2026)

## Abstract (arXiv)

When building AI systems for decision support, one often encounters the phenomenon of predictive multiplicity: a single best model does not exist; instead, one can construct many models with similar overall accuracy that differ in their predictions for individual cases. Especially when decisions have a direct impact on humans, this can be highly unsatisfactory. For a person subject to high disagreement between models, one could as well have chosen a different model of similar overall accuracy that would have decided the person's case differently. We argue that this arbitrariness conflicts with the EU AI Act, which requires providers of high-risk AI systems to report performance not only at the dataset level but also for specific persons. The goal of this paper is to put predictive multiplicity in context with the EU AI Act's provisions on accuracy and to subsequently derive concrete suggestions on how to evaluate and report predictive multiplicity in practice. Specifically: (1) We introduce the AI Act's accuracy provisions and argue that incorporating information about predictive multiplicity could serve compliance with specific provisions for providers. (2) Based on this legally rigorous analysis, we suggest individual conflict ratios and δ-ambiguity as tools to quantify the disagreement between models on individual cases and to help detect individuals subject to conflicting predictions. (3) Based on computational insights, we derive easy-to-implement rules on how model providers could evaluate predictive multiplicity in practice. (4) Ultimately, we suggest that information about predictive multiplicity should be made available to deployers under the AI Act, enabling them to judge whether system outputs for specific individuals are reliable enough for their use case.