# Direct Uncertainty Prediction for Medical Second Opinions

arXiv: https://arxiv.org/abs/1807.01771 · HTML: https://mlsanity.com/paper/1807.01771/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2018.

## Built on by (most cited first, 30 of 61)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Differentiable Learning Under Triage](https://mlsanity.com/paper/2103.08902/) (2021)
- [Towards Unbiased and Accurate Deferral to Multiple Experts](https://mlsanity.com/paper/2102.13004/) (2021)
- [Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithms](https://mlsanity.com/paper/2310.14772/) (2023)
- [Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm](https://mlsanity.com/paper/2207.02942/) (2022)
- [Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstention](https://mlsanity.com/paper/2310.14770/) (2023)
- [Teaching Humans When To Defer to a Classifier via Exemplars](https://mlsanity.com/paper/2111.11297/) (2021)
- [Modality specific U-Net variants for biomedical image segmentation: A survey](https://mlsanity.com/paper/2107.04537/) (2021)
- [Uncertainty Quantification on Clinical Trial Outcome Prediction](https://mlsanity.com/paper/2401.03482/) (2024)
- [Conformalized Credal Regions for Classification with Ambiguous Ground Truth](https://mlsanity.com/paper/2411.04852/) (2024)
- [Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosis](https://mlsanity.com/paper/2108.07392/) (2021)
- [Hyperbolic Uncertainty Aware Semantic Segmentation](https://mlsanity.com/paper/2203.08881/) (2022)
- [CertainNet: Sampling-free Uncertainty Estimation for Object Detection](https://mlsanity.com/paper/2110.01604/) (2021)
- [Counterfactual Inference of Second Opinions](https://mlsanity.com/paper/2203.08653/) (2022)
- [Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration](https://mlsanity.com/paper/2308.03172/) (2023)
- [Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modeling](https://mlsanity.com/paper/2410.11236/) (2024)
- [Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation](https://mlsanity.com/paper/2103.00528/) (2021)
- [Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media](https://mlsanity.com/paper/2305.16797/) (2023)
- [Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints](https://mlsanity.com/paper/2403.06906/) (2024)
- [Designing Closed Human-in-the-loop Deferral Pipelines](https://mlsanity.com/paper/2202.04718/) (2022)
- [A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learning](https://mlsanity.com/paper/2406.00332/) (2024)
- [Disentangling Uncertainty in Machine Translation Evaluation](https://mlsanity.com/paper/2204.06546/) (2022)
- [Harnessing Uncertainty-aware Bounding Boxes for Unsupervised 3D Object Detection](https://mlsanity.com/paper/2408.00619/) (2024)
- [A Semi-Supervised Algorithm for Improving the Consistency of Crowdsourced Datasets: The COVID-19 Case Study on Respiratory Disorder Classification](https://mlsanity.com/paper/2209.04360/) (2022)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)
- [AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems](https://mlsanity.com/paper/2402.06287/) (2024)
- [Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation](https://mlsanity.com/paper/2410.23448/) (2024)
- [Benchmarking Uncertainty Quantification on Biosignal Classification Tasks under Dataset Shift](https://mlsanity.com/paper/2112.09196/) (2021)
- [Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech](https://mlsanity.com/paper/2305.16406/) (2023)
- [Trucks Don't Mean Trump: Diagnosing Human Error in Image Analysis](https://mlsanity.com/paper/2205.07333/) (2022)

## Abstract (arXiv)

The issue of disagreements amongst human experts is a ubiquitous one in both machine learning and medicine. In medicine, this often corresponds to doctor disagreements on a patient diagnosis. In this work, we show that machine learning models can be trained to give uncertainty scores to data instances that might result in high expert disagreements. In particular, they can identify patient cases that would benefit most from a medical second opinion. Our central methodological finding is that Direct Uncertainty Prediction (DUP), training a model to predict an uncertainty score directly from the raw patient features, works better than Uncertainty Via Classification, the two-step process of training a classifier and postprocessing the output distribution to give an uncertainty score. We show this both with a theoretical result, and on extensive evaluations on a large scale medical imaging application.