# Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection

arXiv: https://arxiv.org/abs/2110.14019 · HTML: https://mlsanity.com/paper/2110.14019/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 83 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 30 of 41)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019)
- [Fair Regression: Quantitative Definitions and Reduction-based Algorithms](https://mlsanity.com/paper/1905.12843/) (2019)
- [Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement](https://mlsanity.com/paper/2006.03790/) (2020)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Adversarial Examples Are a Natural Consequence of Test Error in Noise](https://mlsanity.com/paper/1901.10513/) (2019)
- [Hybrid Models for Open Set Recognition](https://mlsanity.com/paper/2003.12506/) (2020)

## Built on by (most cited first, 21 of 21)

- [Test-time Adaptation with Calibration of Medical Image Classification Nets for Label Distribution Shift](https://mlsanity.com/paper/2207.00769/) (2022)
- [Data-SUITE: Data-centric identification of in-distribution incongruous examples](https://mlsanity.com/paper/2202.08836/) (2022)
- [Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets](https://mlsanity.com/paper/2305.09072/) (2023)
- [Explanation Shift: Detecting distribution shifts on tabular data via the explanation space](https://mlsanity.com/paper/2210.12369/) (2022)
- [Explanation Shift: How Did the Distribution Shift Impact the Model?](https://mlsanity.com/paper/2303.08081/) (2023)
- [Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation](https://mlsanity.com/paper/2506.17442/) (2025)
- [Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learning](https://mlsanity.com/paper/2405.12217/) (2024)
- [Temp-SCONE: A Novel Out-of-Distribution Detection and Domain Generalization Framework for Wild Data with Temporal Shift](https://mlsanity.com/paper/2512.04571/) (2025)
- [MLHOps: Machine Learning for Healthcare Operations](https://mlsanity.com/paper/2305.02474/) (2023)
- [Confidence Calibration for Systems with Cascaded Predictive Modules](https://mlsanity.com/paper/2309.12510/) (2023)
- [Out-of-Distribution Detection and Data Drift Monitoring using Statistical Process Control](https://mlsanity.com/paper/2402.08088/) (2024)
- [Prediction of Survival Outcomes under Clinical Presence Shift: A Joint Neural Network Architecture](https://mlsanity.com/paper/2508.05472/) (2025)
- [Assessing Uncertainty Estimation Methods for 3D Image Segmentation under Distribution Shifts](https://mlsanity.com/paper/2402.06937/) (2024)
- [DAGnosis: Localized Identification of Data Inconsistencies using Structures](https://mlsanity.com/paper/2402.17599/) (2024)
- [Self-Supervised Data Generation for Precision Agriculture: Blending Simulated Environments with Real Imagery](https://mlsanity.com/paper/2502.18320/) (2025)
- [Active Multiple-Prediction-Powered Inference](https://mlsanity.com/paper/2605.08429/) (2026)
- [Enhancing Causal Estimation through Unlabeled Offline Data](https://mlsanity.com/paper/2202.07895/) (2022)
- [Diagnostic Tool for Out-of-Sample Model Evaluation](https://mlsanity.com/paper/2206.10982/) (2022)
- [Backprompting: Leveraging Synthetic Production Data for Health Advice Guardrails](https://mlsanity.com/paper/2508.18384/) (2025)
- [Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant care](https://mlsanity.com/paper/2601.08503/) (2026)
- [ProfiliTable: Profiling-Driven Tabular Data Processing via Agentic Workflows](https://mlsanity.com/paper/2605.12376/) (2026)

## Abstract (arXiv)

Unpredictable ML model behavior on unseen data, especially in the health domain, raises serious concerns about its safety as repercussions for mistakes can be fatal. In this paper, we explore the feasibility of using state-of-the-art out-of-distribution detectors for reliable and trustworthy diagnostic predictions. We select publicly available deep learning models relating to various health conditions (e.g., skin cancer, lung sound, and Parkinson's disease) using various input data types (e.g., image, audio, and motion data). We demonstrate that these models show unreasonable predictions on out-of-distribution datasets. We show that Mahalanobis distance- and Gram matrices-based out-of-distribution detection methods are able to detect out-of-distribution data with high accuracy for the health models that operate on different modalities. We then translate the out-of-distribution score into a human interpretable CONFIDENCE SCORE to investigate its effect on the users' interaction with health ML applications. Our user study shows that the textscconfidence score helped the participants only trust the results with a high score to make a medical decision and disregard results with a low score. Through this work, we demonstrate that dataset shift is a critical piece of information for high-stake ML applications, such as medical diagnosis and healthcare, to provide reliable and trustworthy predictions to the users.