# Analyzing the Role of Model Uncertainty for Electronic Health Records

arXiv: https://arxiv.org/abs/1906.03842 · HTML: https://mlsanity.com/paper/1906.03842/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 30 of 41)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration](https://mlsanity.com/paper/2203.13834/) (2022)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [Towards Improving Calibration in Object Detection Under Domain Shift](https://mlsanity.com/paper/2209.07601/) (2022)
- [Reproducibility in Optimization: Theoretical Framework and Limits](https://mlsanity.com/paper/2202.04598/) (2022)
- [Sparse MoEs meet Efficient Ensembles](https://mlsanity.com/paper/2110.03360/) (2021)
- [Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation](https://mlsanity.com/paper/2107.04212/) (2021)
- [Multiclass Confidence and Localization Calibration for Object Detection](https://mlsanity.com/paper/2306.08271/) (2023)
- [Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration](https://mlsanity.com/paper/2308.03172/) (2023)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection](https://mlsanity.com/paper/2303.14404/) (2023)
- [Cal-DETR: Calibrated Detection Transformer](https://mlsanity.com/paper/2311.03570/) (2023)
- [Synthesizing Irreproducibility in Deep Networks](https://mlsanity.com/paper/2102.10696/) (2021)
- [Beyond Classification: Definition and Density-based Estimation of Calibration in Object Detection](https://mlsanity.com/paper/2312.06645/) (2023)
- [UncertaintyFuseNet: Robust Uncertainty-aware Hierarchical Feature Fusion Model with Ensemble Monte Carlo Dropout for COVID-19 Detection](https://mlsanity.com/paper/2105.08590/) (2021)
- [Deep Classifiers with Label Noise Modeling and Distance Awareness](https://mlsanity.com/paper/2110.02609/) (2021)
- [Evaluating subgroup disparity using epistemic uncertainty in mammography](https://mlsanity.com/paper/2107.02716/) (2021)
- [Maximum Likelihood Uncertainty Estimation: Robustness to Outliers](https://mlsanity.com/paper/2202.03870/) (2022)
- [Uncertainty-aware abstention in medical diagnosis based on medical texts](https://mlsanity.com/paper/2502.18050/) (2025)
- [Disentangling Model Multiplicity in Deep Learning](https://mlsanity.com/paper/2206.08890/) (2022)
- [Uncertainty-Aware Boosted Ensembling in Multi-Modal Settings](https://mlsanity.com/paper/2104.10715/) (2021)
- [BEDS-Bench: Behavior of EHR-models under Distributional Shift--A Benchmark](https://mlsanity.com/paper/2107.08189/) (2021)
- [Uncertainty-Aware Text-to-Program for Question Answering on Structured Electronic Health Records](https://mlsanity.com/paper/2203.06918/) (2022)
- [Empirical Optimal Risk to Quantify Model Trustworthiness for Failure Detection](https://mlsanity.com/paper/2308.03179/) (2023)
- [Bayesian Neural Hawkes Process for Event Uncertainty Prediction](https://mlsanity.com/paper/2112.14474/) (2021)
- [Compound Density Networks for Risk Prediction using Electronic Health Records](https://mlsanity.com/paper/2208.01320/) (2022)

## Abstract (arXiv)

In medicine, both ethical and monetary costs of incorrect predictions can be significant, and the complexity of the problems often necessitates increasingly complex models. Recent work has shown that changing just the random seed is enough for otherwise well-tuned deep neural networks to vary in their individual predicted probabilities. In light of this, we investigate the role of model uncertainty methods in the medical domain. Using RNN ensembles and various Bayesian RNNs, we show that population-level metrics, such as AUC-PR, AUC-ROC, log-likelihood, and calibration error, do not capture model uncertainty. Meanwhile, the presence of significant variability in patient-specific predictions and optimal decisions motivates the need for capturing model uncertainty. Understanding the uncertainty for individual patients is an area with clear clinical impact, such as determining when a model decision is likely to be brittle. We further show that RNNs with only Bayesian embeddings can be a more efficient way to capture model uncertainty compared to ensembles, and we analyze how model uncertainty is impacted across individual input features and patient subgroups.