# Learning under selective labels in the presence of expert consistency

arXiv: https://arxiv.org/abs/1807.00905 · HTML: https://mlsanity.com/paper/1807.00905/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2018.

## Built on by (most cited first, 18 of 18)

- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [Fairness-Aware PAC Learning from Corrupted Data](https://mlsanity.com/paper/2102.06004/) (2021)
- [Leveraging Expert Consistency to Improve Algorithmic Decision Support](https://mlsanity.com/paper/2101.09648/) (2021)
- [Human-AI Collaboration in Decision-Making: Beyond Learning to Defer](https://mlsanity.com/paper/2206.13202/) (2022)
- [Learning Personalized Decision Support Policies](https://mlsanity.com/paper/2304.06701/) (2023)
- [Adaptive Data Debiasing through Bounded Exploration](https://mlsanity.com/paper/2110.13054/) (2021)
- [Individually Fair Learning with One-Sided Feedback](https://mlsanity.com/paper/2206.04475/) (2022)
- [Algorithmic Censoring in Dynamic Learning Systems](https://mlsanity.com/paper/2305.09035/) (2023)
- [Prediction without Preclusion: Recourse Verification with Reachable Sets](https://mlsanity.com/paper/2308.12820/) (2023)
- [Learning with Selectively Labeled Data from Multiple Decision-makers](https://mlsanity.com/paper/2306.07566/) (2023)
- [Monotone Individual Fairness](https://mlsanity.com/paper/2403.06812/) (2024)
- [Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest](https://mlsanity.com/paper/2504.04243/) (2025)
- [Fair Classification with Partial Feedback: An Exploration-Based Data Collection Approach](https://mlsanity.com/paper/2402.11338/) (2024)
- [Predictive Performance Comparison of Decision Policies Under Confounding](https://mlsanity.com/paper/2404.00848/) (2024)
- [HR-Bandit: Human-AI Collaborated Linear Recourse Bandit](https://mlsanity.com/paper/2410.14640/) (2024)
- [Fairness under uncertainty in sequential decisions](https://mlsanity.com/paper/2604.21711/) (2026)
- [Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias](https://mlsanity.com/paper/2606.14506/) (2026)
- [Cost Efficient Fairness Audit Under Partial Feedback](https://mlsanity.com/paper/2510.03734/) (2025)

## Abstract (arXiv)

We explore the problem of learning under selective labels in the context of algorithm-assisted decision making. Selective labels is a pervasive selection bias problem that arises when historical decision making blinds us to the true outcome for certain instances. Examples of this are common in many applications, ranging from predicting recidivism using pre-trial release data to diagnosing patients. In this paper we discuss why selective labels often cannot be effectively tackled by standard methods for adjusting for sample selection bias, even if there are no unobservables. We propose a data augmentation approach that can be used to either leverage expert consistency to mitigate the partial blindness that results from selective labels, or to empirically validate whether learning under such framework may lead to unreliable models prone to systemic discrimination.