# Benchmarking Uncertainty Quantification on Biosignal Classification Tasks under Dataset Shift

arXiv: https://arxiv.org/abs/2112.09196 · HTML: https://mlsanity.com/paper/2112.09196/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 23 of 23)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Confidence-Aware Learning for Deep Neural Networks](https://mlsanity.com/paper/2007.01458/) (2020)
- [Direct Uncertainty Prediction for Medical Second Opinions](https://mlsanity.com/paper/1807.01771/) (2018)
- [Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data](https://mlsanity.com/paper/2006.05919/) (2020)
- [Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification](https://mlsanity.com/paper/1909.01940/) (2019)
- [Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data](https://mlsanity.com/paper/2011.03274/) (2020)
- [AI4COVID-19: AI Enabled Preliminary Diagnosis for COVID-19 from Cough Samples via an App](https://mlsanity.com/paper/2004.01275/) (2020)
- [Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection](https://mlsanity.com/paper/2003.10769/) (2020)
- [Accurate Uncertainty Estimation and Decomposition in Ensemble Learning](https://mlsanity.com/paper/1911.04061/) (2019)
- [RespireNet: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting](https://mlsanity.com/paper/2011.00196/) (2020)
- [Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?](https://mlsanity.com/paper/2011.03178/) (2020)
- [Sounds of COVID-19: exploring realistic performance of audio-based digital testing](https://mlsanity.com/paper/2106.15523/) (2021)
- [Uncertainty aware and explainable diagnosis of retinal disease](https://mlsanity.com/paper/2101.12041/) (2021)
- [Uncertainty-Aware COVID-19 Detection from Imbalanced Sound Data](https://mlsanity.com/paper/2104.02005/) (2021)

## Built on by (most cited first, 6 of 6)

- [Uncertainty Quantification in Machine Learning for Biosignal Applications -- A Review](https://mlsanity.com/paper/2312.09454/) (2023)
- [Informative Priors Improve the Reliability of Multimodal Clinical Data Classification](https://mlsanity.com/paper/2312.00794/) (2023)
- [ECG-Based Electrolyte Prediction: Evaluating Regression and Probabilistic Methods](https://mlsanity.com/paper/2212.13890/) (2022)
- [Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift](https://mlsanity.com/paper/2409.03543/) (2024)
- [Data-Driven Priors for Uncertainty-Aware Risk Prediction of Clinical Deterioration using Multimodal Data](https://mlsanity.com/paper/2603.08459/) (2026)
- [ADAPTOOD: Uncertainty-Aware Fine-Tuning for Out-of-Distribution ECG Time Series Models](https://mlsanity.com/paper/2606.04164/) (2026)

## Abstract (arXiv)

A biosignal is a signal that can be continuously measured from human bodies, such as respiratory sounds, heart activity (ECG), brain waves (EEG), etc, based on which, machine learning models have been developed with very promising performance for automatic disease detection and health status monitoring. However, dataset shift, i.e., data distribution of inference varies from the distribution of the training, is not uncommon for real biosignal-based applications. To improve the robustness, probabilistic models with uncertainty quantification are adapted to capture how reliable a prediction is. Yet, assessing the quality of the estimated uncertainty remains a challenge. In this work, we propose a framework to evaluate the capability of the estimated uncertainty in capturing different types of biosignal dataset shifts with various degrees. In particular, we use three classification tasks based on respiratory sounds and electrocardiography signals to benchmark five representative uncertainty quantification methods. Extensive experiments show that, although Ensemble and Bayesian models could provide relatively better uncertainty estimations under dataset shifts, all tested models fail to meet the promise in trustworthy prediction and model calibration. Our work paves the way for a comprehensive evaluation for any newly developed biosignal classifiers.