mlsanityData to 14 Oct 2026

Exploring How Anomalous Model Input and Output Alerts Affect Decision-Making in Healthcare

arXiv 2204.13194 ↗First posted 27 Apr 2022cs.HC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

An important goal in the field of human-AI interaction is to help users more appropriately trust AI systems' decisions. A situation in which the user may particularly benefit from more appropriate trust is when the AI receives anomalous input or provides anomalous output. To the best of our knowledge, this is the first work towards understanding how anomaly alerts may contribute to appropriate trust of AI. In a formative mixed-methods study with 4 radiologists and 4 other physicians, we explore how AI alerts for anomalous input, very high and low confidence, and anomalous saliency-map explanations affect users' experience with mockups of an AI clinical decision support system (CDSS) for evaluating chest x-rays for pneumonia. We find evidence suggesting that the four anomaly alerts are desired by non-radiologists, and the high-confidence alerts are desired by both radiologists and non-radiologists. In a follow-up user study, we investigate how high- and low-confidence alerts affect the accuracy and thus appropriate trust of 33 radiologists working with AI CDSS mockups. We observe that these alerts do not improve users' accuracy or experience and discuss potential reasons why.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 80 references that are in the map. First posted on arXiv 27 Apr 2022.

Builds on

33 of 80 references.

References · 33

  1. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
  2. 2019Attention is not Explanationcited by 929
  3. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  4. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 453
  5. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  6. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  7. 2018Learning Confidence for Out-of-Distribution Detection in Neural Networkscited by 245
  8. 2019What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Usecited by 135
  9. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  10. 2017What do we need to build explainable AI systems for the medical domain?cited by 129
  11. 2020Explainable Deep One-Class Classificationcited by 127
  12. 2021XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanationscited by 73
21 more
  1. 2020On the limits of cross-domain generalization in automated X-ray predictioncited by 62
  2. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  3. 2019Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processescited by 53
  4. 2019Explainable artificial intelligence model to predict acute critical illness from electronic health recordscited by 42
  5. 2020CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysiscited by 40
  6. 2020A Benchmark of Medical Out of Distribution Detectioncited by 40
  7. 2021Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lenscited by 35
  8. 2021Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysiscited by 34
  9. 2020OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samplescited by 33
  10. 2020A Critic Evaluation of Methods for COVID-19 Automatic Detection from X-Ray Imagescited by 28
  11. 2020Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Makingcited by 25
  12. 2019A Human-Grounded Evaluation of SHAP for Alert Processingcited by 23
  13. 2021Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutionscited by 16
  14. 2020Impact of lung segmentation on the diagnosis and explanation of COVID-19 in chest X-ray imagescited by 14
  15. 2019Chester: A Web Delivered Locally Computed Chest X-Ray Disease Prediction Systemcited by 12
  16. 2021Resisting Out-of-Distribution Data Problem in Perturbation of XAIcited by 12
  17. 2020DeepCOVIDExplainer: Explainable COVID-19 Diagnosis Based on Chest X-ray Imagescited by 11
  18. 2020Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Datasetcited by 7
  19. 2021Explainable AI, but explainable to whom?cited by 7
  20. 2021Comparing radiologists' gaze and saliency maps generated by interpretability methods for chest x-rayscited by 7
  21. 2021Structured dataset documentation: a datasheet for CheXpertcited by 6

Built on by

Not cited yet.