mlsanityData to 7 Oct 2026

Direct Uncertainty Prediction for Medical Second Opinions

arXiv 1807.01771 ↗First posted 4 Jul 2018cs.LG stat.MLcited by 61

Not on the map. Why?

Abstract (arXiv)

The issue of disagreements amongst human experts is a ubiquitous one in both machine learning and medicine. In medicine, this often corresponds to doctor disagreements on a patient diagnosis. In this work, we show that machine learning models can be trained to give uncertainty scores to data instances that might result in high expert disagreements. In particular, they can identify patient cases that would benefit most from a medical second opinion. Our central methodological finding is that Direct Uncertainty Prediction (DUP), training a model to predict an uncertainty score directly from the raw patient features, works better than Uncertainty Via Classification, the two-step process of training a classifier and postprocessing the output distribution to give an uncertainty score. We show this both with a theoretical result, and on extensive evaluations on a large scale medical imaging application.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 13; 2023: 14; 2024: 16; 2025: 3; 2026: 5.

61 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2021Differentiable Learning Under Triagecited by 74
  3. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
  4. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  5. 2022Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithmcited by 35
  6. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  7. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  8. 2021Modality specific U-Net variants for biomedical image segmentation: A surveycited by 20
  9. 2024Uncertainty Quantification on Clinical Trial Outcome Predictioncited by 19
  10. 2024Conformalized Credal Regions for Classification with Ambiguous Ground Truthcited by 19
  11. 2022Hyperbolic Uncertainty Aware Semantic Segmentationcited by 17
  12. 2021Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosiscited by 15
38 more
  1. 2021CertainNet: Sampling-free Uncertainty Estimation for Object Detectioncited by 15
  2. 2022Counterfactual Inference of Second Opinionscited by 14
  3. 2023Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibrationcited by 14
  4. 2024Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modelingcited by 14
  5. 2021Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimationcited by 13
  6. 2023Calibration of Transformer-based Models for Identifying Stress and Depression in Social Mediacited by 12
  7. 2024Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraintscited by 12
  8. 2022Designing Closed Human-in-the-loop Deferral Pipelinescited by 11
  9. 2022Disentangling Uncertainty in Machine Translation Evaluationcited by 9
  10. 2024A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learningcited by 9
  11. 2024Harnessing Uncertainty-aware Bounding Boxes for Unsupervised 3D Object Detectioncited by 9
  12. 2022A Semi-Supervised Algorithm for Improving the Consistency of Crowdsourced Datasets: The COVID-19 Case Study on Respiratory Disorder Classificationcited by 7
  13. 2023Evaluating AI systems under uncertain ground truth: a case study in dermatologycited by 7
  14. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  15. 2024Venire: A Machine Learning-Guided Panel Review System for Community Content Moderationcited by 7
  16. 2021Benchmarking Uncertainty Quantification on Biosignal Classification Tasks under Dataset Shiftcited by 6
  17. 2023Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speechcited by 5
  18. 2022Trucks Don't Mean Trump: Diagnosing Human Error in Image Analysiscited by 4
  19. 2023Empirical Optimal Risk to Quantify Model Trustworthiness for Failure Detectioncited by 4
  20. 2023When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-makingcited by 4
  21. 2025How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyondcited by 4
  22. 2022TEDL: A Two-stage Evidential Deep Learning Method for Classification Uncertainty Quantificationcited by 3
  23. 2022An Exploratory Study of AI System Risk Assessment from the Lens of Data Distribution and Uncertaintycited by 3
  24. 2023FiFAR: A Fraud Detection Dataset for Learning to Defercited by 3
  25. 2025Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)cited by 3
  26. 2021Rethinking Transfer Learning for Medical Image Classificationcited by 2
  27. 2022Doubting AI Predictions: Influence-Driven Second Opinion Recommendationcited by 2
  28. 2023Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applicationscited by 2
  29. 2023Delay-SDE-net: A deep learning approach for time series modelling with memory and uncertainty estimatescited by 2
  30. 2024Reliability and Interpretability in Science and Deep Learningcited by 2
  31. 2024Context-aware Multi-Model Object Detection for Diversely Heterogeneous Compute Systemscited by 2
  32. 2024Visual Analysis of Prediction Uncertainty in Neural Networks for Deep Image Synthesiscited by 2
  33. 2024WildFit: Autonomous In-situ Model Adaptation for Resource-Constrained IoT Systemscited by 2
  34. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Modelscited by 2
  35. 2022Label uncertainty-guided multi-stream model for disease screeningcited by 1
  36. 2022IDPS Signature Classification with a Reject Option and the Incorporation of Expert Knowledgecited by 1
  37. 2023Statistical Tests for Replacing Human Decision Makers with Algorithmscited by 1
  38. 2026Decomposing Physician Disagreement in HealthBenchcited by 1

50 most cited of 61 All 61 (JSON)