Towards Multi-Stakeholder Evaluation of ML Models: A Crowdsourcing Study on Metric Preferences in Job-matching System
Not on the map. Why?
Abstract (arXiv)
While machine learning (ML) technology affects diverse stakeholders, there is no one-size-fits-all metric to evaluate the quality of outputs, including performance and fairness. Using predetermined metrics without soliciting stakeholder opinions is problematic because it leads to an unfair disregard for stakeholders in the ML pipeline. In this study, to establish practical ways to incorporate diverse stakeholder opinions into the selection of metrics for ML, we investigate participants' preferences for different metrics by using crowdsourcing. We ask 837 participants to choose a better model from two hypothetical ML models in a hypothetical job-matching system twenty times and calculate their utility values for seven metrics. To examine the participants' feedback in detail, we divide them into five clusters based on their utility values and analyze the tendencies of each cluster, including their preferences for metrics and common attributes. Based on the results, we discuss the points that should be considered when selecting appropriate metrics and evaluating ML models with multiple stakeholders.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 12 of its 42 references that are in the map. First posted on arXiv 3 Mar 2025.
Builds on
12 of 42 references.
References · 12
- 2016Equality of Opportunity in Supervised Learningcited by 2,007
- 2018Model Cards for Model Reportingcited by 1,080
- 2017Counterfactual Fairnesscited by 804
- 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 672
- 2020Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlationcited by 425
- 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
- 2019FairSight: Visual Analytics for Fairness in Decision Makingcited by 59
- 2019Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learningcited by 57
- 2018Stakeholders in Explainable AIcited by 48
- 2020Empirical observation of negligible fairness-accuracy trade-offs in machine learning for public policycited by 44
- 2021Soliciting Stakeholders' Fairness Notions in Child Maltreatment Predictive Systemscited by 27
- 2022Towards Responsible AI: A Design Space Exploration of Human-Centered Artificial Intelligence User Interfaces to Investigate Fairnesscited by 9
Built on by
Citing papers per year: 2025: 1.
1 paper.