# Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest

arXiv: https://arxiv.org/abs/2504.04243 · HTML: https://mlsanity.com/paper/2504.04243/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 86 references that are in the map.
- First posted on arXiv 5 Apr 2025.

## Builds on (most cited first, 30 of 32)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [Identifying and Correcting Label Bias in Machine Learning](https://mlsanity.com/paper/1901.04966/) (2019)
- [Expanding Explainability: Towards Social Transparency in AI systems](https://mlsanity.com/paper/2101.04719/) (2021)
- [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021)
- [Jury Learning: Integrating Dissenting Voices into Machine Learning Models](https://mlsanity.com/paper/2202.02950/) (2022)
- [Problem Formulation and Fairness](https://mlsanity.com/paper/1901.02547/) (2019)
- [Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations](https://mlsanity.com/paper/2302.02187/) (2023)
- [Who Should Predict? Exact Algorithms For Learning to Defer to Humans](https://mlsanity.com/paper/2301.06197/) (2023)
- [Characterizing Fairness Over the Set of Good Models Under Selective Labels](https://mlsanity.com/paper/2101.00352/) (2021)
- [In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making](https://mlsanity.com/paper/2305.07722/) (2023)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems](https://mlsanity.com/paper/2301.11333/) (2023)
- [Fairness Evaluation in Presence of Biased Noisy Labels](https://mlsanity.com/paper/2003.13808/) (2020)
- [Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis](https://mlsanity.com/paper/2309.12368/) (2023)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making](https://mlsanity.com/paper/2209.11812/) (2022)
- [Toward Supporting Perceptual Complementarity in Human-AI Collaboration via Reflection on Unobservables](https://mlsanity.com/paper/2207.13834/) (2022)
- [Learning under selective labels in the presence of expert consistency](https://mlsanity.com/paper/1807.00905/) (2018)
- [Leveraging Expert Consistency to Improve Algorithmic Decision Support](https://mlsanity.com/paper/2101.09648/) (2021)
- [Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making](https://mlsanity.com/paper/2401.07058/) (2024)
- [Crowdsourcing Ground Truth for Medical Relation Extraction](https://mlsanity.com/paper/1701.02185/) (2017)
- [AI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of Interventions](https://mlsanity.com/paper/2304.08804/) (2023)

## Built on by (most cited first, 3 of 3)

- [Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias](https://mlsanity.com/paper/2606.14506/) (2026)
- [Label Indeterminacy in AI & Law](https://mlsanity.com/paper/2510.17463/) (2025)
- [Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication](https://mlsanity.com/paper/2608.12477/) (2026)

## Abstract (arXiv)

The design of AI systems to assist human decision-making typically requires the availability of labels to train and evaluate supervised models. Frequently, however, these labels are unknown, and different ways of estimating them involve unverifiable assumptions or arbitrary choices. In this work, we introduce the concept of label indeterminacy and derive important implications in high-stakes AI-assisted decision-making. We present an empirical study in a healthcare context, focusing specifically on predicting the recovery of comatose patients after resuscitation from cardiac arrest. Our study shows that label indeterminacy can result in models that perform similarly when evaluated on patients with known labels, but vary drastically in their predictions for patients where labels are unknown. After demonstrating crucial ethical implications of label indeterminacy in this high-stakes context, we discuss takeaways for evaluation, reporting, and design.