mlsanityData to 14 Oct 2026

Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)

arXiv 2505.11738 ↗First posted 16 May 2025cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) tools for radiology are commonly unmonitored once deployed. The lack of real-time case-by-case assessments of AI prediction confidence requires users to independently distinguish between trustworthy and unreliable AI predictions, which increases cognitive burden, reduces productivity, and potentially leads to misdiagnoses. To address these challenges, we introduce Ensembled Monitoring Model (EMM), a framework inspired by clinical consensus practices using multiple expert reviews. Designed specifically for black-box commercial AI products, EMM operates independently without requiring access to internal AI components or intermediate outputs, while still providing robust confidence measurements. Using intracranial hemorrhage detection as our test case on a large, diverse dataset of 2919 studies, we demonstrate that EMM successfully categorizes confidence in the AI-generated prediction, suggesting different actions and helping improve the overall performance of AI tools to ultimately reduce cognitive burden. Importantly, we provide key technical considerations and best practices for successfully translating EMM into clinical settings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 59 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

29 of 59 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  11. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  12. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
17 more
  1. 2022MONAI: An open-source framework for deep learning in healthcarecited by 448
  2. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  3. 2019Verified Uncertainty Calibrationcited by 309
  4. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  5. 2019Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentationcited by 163
  6. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  7. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 150
  8. 2024Merlin: A Computed Tomography Vision-Language Foundation Model and Datasetcited by 102
  9. 2024A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretationcited by 81
  10. 2022Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysiscited by 63
  11. 2018Direct Uncertainty Prediction for Medical Second Opinionscited by 61
  12. 2021AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantificationcited by 31
  13. 2019Density estimation in representation space to predict model uncertaintycited by 27
  14. 2023The state-of-the-art 3D anisotropic intracranial hemorrhage segmentation on non-contrast head CT: The INSTANCE challengecited by 26
  15. 2022Improving the repeatability of deep learning models with Monte Carlo dropoutcited by 11
  16. 2022Voxels Intersecting along Orthogonal Levels Attention U-Net for Intracerebral Haemorrhage Segmentation in Head CTcited by 4
  17. 2024SUDO: a framework for evaluating clinical artificial intelligence systems without ground-truth annotationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Pillar-0: A New Frontier for Radiology Foundation Modelscited by 25
  2. 2025A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Toolcited by 0
  3. 2026Improving Calibration of Black-Box Radiology AI Using Test-Time Augmentationcited by 0