# How Much Imprecision is Enough Imprecision in my Classifier? A Practical Elicitation Procedure

arXiv: https://arxiv.org/abs/2609.33352 · HTML: https://mlsanity.com/paper/2609.33352/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 8 of 8)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rules](https://mlsanity.com/paper/2404.12215/) (2024)
- [Fair Performance Metric Elicitation](https://mlsanity.com/paper/2006.12732/) (2020)
- [Scoring Rules and Calibration for Imprecise Probabilities](https://mlsanity.com/paper/2410.23001/) (2024)
- [Performance Metric Elicitation from Pairwise Classifier Comparisons](https://mlsanity.com/paper/1806.01827/) (2018)
- [Truthful Elicitation of Imprecise Forecasts](https://mlsanity.com/paper/2503.16395/) (2025)

## Abstract (arXiv)

Set-valued classifiers, whether derived from precise probabilities and an adapted cost function, from convex sets with a robust inference mechanism, or from conformal methods, are routine options to obtain more robust, trustworthy predictions. However, there is a lack of operational tools to measure how robust or imprecise a given user is ready to be when receiving predictions, that is how much precision he/she is ready to let go in exchange of more accuracy. This is why we propose, in this paper, practical and operational elicitation procedures to measure the user proneness to set-valued predictions. The effectiveness of the iterative elicitation procedure in converging to the target parameter value is demonstrated on both tabular and image datasets drawn from standard machine learning benchmarks. The results show that the procedure also presents the user with a small number of instances, highlighting the practicality of the approach for real-world applications aimed at identifying the decision maker's optimal behavior when faced with imprecision.