# Metric Elicitation; Moving from Theory to Practice

arXiv: https://arxiv.org/abs/2212.03495 · HTML: https://mlsanity.com/paper/2212.03495/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 16 references that are in the map.
- First posted on arXiv 7 Dec 2022.

## Builds on (most cited first, 4 of 4)

- [Fair Performance Metric Elicitation](https://mlsanity.com/paper/2006.12732/) (2020)
- [Joint Optimization of AI Fairness and Utility: A Human-Centered Approach](https://mlsanity.com/paper/2002.01621/) (2020)
- [Performance Metric Elicitation from Pairwise Classifier Comparisons](https://mlsanity.com/paper/1806.01827/) (2018)
- [Quadratic Metric Elicitation for Fairness and Beyond](https://mlsanity.com/paper/2011.01516/) (2020)

## Abstract (arXiv)

Metric Elicitation (ME) is a framework for eliciting classification metrics that better align with implicit user preferences based on the task and context. The existing ME strategy so far is based on the assumption that users can most easily provide preference feedback over classifier statistics such as confusion matrices. This work examines ME, by providing a first ever implementation of the ME strategy. Specifically, we create a web-based ME interface and conduct a user study that elicits users' preferred metrics in a binary classification setting. We discuss the study findings and present guidelines for future research in this direction.