# Leveraging Expert Consistency to Improve Algorithmic Decision Support

arXiv: https://arxiv.org/abs/2101.09648 · HTML: https://mlsanity.com/paper/2101.09648/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 83 references that are in the map.
- First posted on arXiv 24 Jan 2021.

## Builds on (most cited first, 16 of 16)

- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [Problem Formulation and Fairness](https://mlsanity.com/paper/1901.02547/) (2019)
- [A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores](https://mlsanity.com/paper/2002.08035/) (2020)
- [Discrimination in the Age of Algorithms](https://mlsanity.com/paper/1902.03731/) (2019)
- [An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?](https://mlsanity.com/paper/2011.14999/) (2020)
- [A Human-Centered Review of the Algorithms used within the U.S. Child Welfare System](https://mlsanity.com/paper/2003.03541/) (2020)
- [Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts](https://mlsanity.com/paper/2012.05370/) (2020)
- [Learning under selective labels in the presence of expert consistency](https://mlsanity.com/paper/1807.00905/) (2018)
- [Learning Credible Models](https://mlsanity.com/paper/1711.03190/) (2017)
- [Fair Machine Learning Under Partial Compliance](https://mlsanity.com/paper/2011.03654/) (2020)

## Built on by (most cited first, 17 of 17)

- [Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support](https://mlsanity.com/paper/2204.02310/) (2022)
- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [Toward Supporting Perceptual Complementarity in Human-AI Collaboration via Reflection on Unobservables](https://mlsanity.com/paper/2207.13834/) (2022)
- [User-Interactive Offline Reinforcement Learning](https://mlsanity.com/paper/2205.10629/) (2022)
- [Counterfactual Prediction Under Outcome Measurement Error](https://mlsanity.com/paper/2302.11121/) (2023)
- [Domain constraints improve risk prediction when outcome data is missing](https://mlsanity.com/paper/2312.03878/) (2023)
- [Perspectives on Incorporating Expert Feedback into Model Updates](https://mlsanity.com/paper/2205.06905/) (2022)
- [Learning Complementary Policies for Human-AI Teams](https://mlsanity.com/paper/2302.02944/) (2023)
- [Quantifying and mitigating the impact of label errors on model disparity metrics](https://mlsanity.com/paper/2310.02533/) (2023)
- [More Data Can Lead Us Astray: Active Data Acquisition in the Presence of Label Bias](https://mlsanity.com/paper/2207.07723/) (2022)
- [Learning with Selectively Labeled Data from Multiple Decision-makers](https://mlsanity.com/paper/2306.07566/) (2023)
- [Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest](https://mlsanity.com/paper/2504.04243/) (2025)
- [Doubting AI Predictions: Influence-Driven Second Opinion Recommendation](https://mlsanity.com/paper/2205.00072/) (2022)
- [Mitigating Label Bias via Decoupled Confident Learning](https://mlsanity.com/paper/2307.08945/) (2023)
- [SCRuB: Social Concept Reasoning under Rubric-Based Evaluation](https://mlsanity.com/paper/2605.06444/) (2026)
- [Recentering Validity Considerations through Early-Stage Deliberations Around AI and Policy Design](https://mlsanity.com/paper/2303.14602/) (2023)
- [Gated Decoupled Compositional Bandits: A Unified Theory of Contextual Bandits with Supervised-Calibrated Action Scaling and Pre-Execution Gating](https://mlsanity.com/paper/2608.21993/) (2026)

## Abstract (arXiv)

Machine learning (ML) is increasingly being used to support high-stakes decisions. However, there is frequently a construct gap: a gap between the construct of interest to the decision-making task and what is captured in proxies used as labels to train ML models. As a result, ML models may fail to capture important dimensions of decision criteria, hampering their utility for decision support. Thus, an essential step in the design of ML systems for decision support is selecting a target label among available proxies. In this work, we explore the use of historical expert decisions as a rich -- yet also imperfect -- source of information that can be combined with observed outcomes to narrow the construct gap. We argue that managers and system designers may be interested in learning from experts in instances where they exhibit consistency with each other, while learning from observed outcomes otherwise. We develop a methodology to enable this goal using information that is commonly available in organizational information systems. This involves two core steps. First, we propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Second, we introduce a label amalgamation approach that allows ML models to simultaneously learn from expert decisions and observed outcomes. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap, yielding better predictive performance than learning from either observed outcomes or expert decisions alone.