# Fair Performance Metric Elicitation

arXiv: https://arxiv.org/abs/2006.12732 · HTML: https://mlsanity.com/paper/2006.12732/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2020.

## Built on by (most cited first, 13 of 13)

- [A Sociotechnical View of Algorithmic Fairness](https://mlsanity.com/paper/2110.09253/) (2021)
- [Moral Machine or Tyranny of the Majority?](https://mlsanity.com/paper/2305.17319/) (2023)
- [Uncertain Decisions Facilitate Better Preference Learning](https://mlsanity.com/paper/2106.10394/) (2021)
- [Perspectives on Incorporating Expert Feedback into Model Updates](https://mlsanity.com/paper/2205.06905/) (2022)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [Optimizing Black-box Metrics with Iterative Example Weighting](https://mlsanity.com/paper/2102.09492/) (2021)
- [Superhuman Fairness](https://mlsanity.com/paper/2301.13420/) (2023)
- [A Note on Bias to Complete](https://mlsanity.com/paper/2402.11710/) (2024)
- [Metric Elicitation; Moving from Theory to Practice](https://mlsanity.com/paper/2212.03495/) (2022)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [An Efficient Plugin Method for Metric Optimization of Black-Box Models](https://mlsanity.com/paper/2503.02119/) (2025)
- ["I think this is fair": Uncovering the Complexities of Stakeholder Decision-Making in AI Fairness Assessment](https://mlsanity.com/paper/2509.17956/) (2025)
- [How Much Imprecision is Enough Imprecision in my Classifier? A Practical Elicitation Procedure](https://mlsanity.com/paper/2609.33352/) (2026)

## Abstract (arXiv)

What is a fair performance metric? We consider the choice of fairness metrics through the lens of metric elicitation -- a principled framework for selecting performance metrics that best reflect implicit preferences. The use of metric elicitation enables a practitioner to tune the performance and fairness metrics to the task, context, and population at hand. Specifically, we propose a novel strategy to elicit group-fair performance metrics for multiclass classification problems with multiple sensitive groups that also includes selecting the trade-off between predictive performance and fairness violation. The proposed elicitation strategy requires only relative preference feedback and is robust to both finite sample and feedback noise.