# Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-making arXiv: https://arxiv.org/abs/2204.05030 ยท HTML: https://mlsanity.com/paper/2204.05030/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 81 references that are in the map. - First posted on arXiv 11 Apr 2022. ## Builds on (most cited first, 24 of 24) - [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017) - [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018) - [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016) - [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018) - [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019) - [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019) - [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020) - [Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives](https://mlsanity.com/paper/1802.07623/) (2018) - [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018) - [Interpretable Counterfactual Explanations Guided by Prototypes](https://mlsanity.com/paper/1907.02584/) (2019) - [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016) - [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018) - [Problems with Shapley-value-based explanations as feature importance measures](https://mlsanity.com/paper/2002.11097/) (2020) - [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021) - ["How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations](https://mlsanity.com/paper/1911.06473/) (2019) - [Towards Explainable Artificial Intelligence](https://mlsanity.com/paper/1909.12072/) (2019) - [Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes](https://mlsanity.com/paper/1904.09612/) (2019) - [Does Explainable Artificial Intelligence Improve Human Decision-Making?](https://mlsanity.com/paper/2006.11194/) (2020) - [A Survey on Explainability in Machine Reading Comprehension](https://mlsanity.com/paper/2010.00389/) (2020) - [Making Tree Ensembles Interpretable: A Bayesian Model Selection Approach](https://mlsanity.com/paper/1606.09066/) (2016) - [A Human-Grounded Evaluation of SHAP for Alert Processing](https://mlsanity.com/paper/1907.03324/) (2019) - [Learning Functional Causal Models with Generative Neural Networks](https://mlsanity.com/paper/1709.05321/) (2017) - [When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making](https://mlsanity.com/paper/2011.06167/) (2020) - [ClinicalVis: Supporting Clinical Task-Focused Design Evaluation](https://mlsanity.com/paper/1810.05798/) (2018) ## Built on by (most cited first, 14 of 14) - [Probabilistic Medical Predictions of Large Language Models](https://mlsanity.com/paper/2408.11316/) (2024) - [A systematic review of biologically-informed deep learning models for cancer: fundamental trends for encoding and interpreting oncology data](https://mlsanity.com/paper/2207.00812/) (2022) - [A Survey of AI Reliance](https://mlsanity.com/paper/2408.03948/) (2024) - [Measuring Perceived Trust in XAI-Assisted Decision-Making by Eliciting a Mental Model](https://mlsanity.com/paper/2307.11765/) (2023) - [Hierarchical Salient Patch Identification for Interpretable Fundus Disease Localization](https://mlsanity.com/paper/2405.14334/) (2024) - [Not someone, but something: Rethinking trust in the age of medical AI](https://mlsanity.com/paper/2504.05331/) (2025) - [Exploring the Effect of Explanation Content and Format on User Comprehension and Trust in Healthcare](https://mlsanity.com/paper/2408.17401/) (2024) - [Diagnosis of Malignant Lymphoma Cancer Using Hybrid Optimized Techniques Based on Dense Neural Networks](https://mlsanity.com/paper/2410.06974/) (2024) - [Contrasting Attitudes Towards Current and Future AI Applications for Computerised Interpretation of ECG: A Clinical Stakeholder Interview Study](https://mlsanity.com/paper/2410.16879/) (2024) - [REMEMBER: Retrieval-based Explainable Multimodal Evidence-guided Modeling for Brain Evaluation and Reasoning in Zero- and Few-shot Neurodegenerative Diagnosis](https://mlsanity.com/paper/2504.09354/) (2025) - [Explainability of Algorithms](https://mlsanity.com/paper/2508.13529/) (2025) - [Towards a Transparent and Interpretable AI Model for Medical Image Classifications](https://mlsanity.com/paper/2509.16685/) (2025) - [From Clinical Intent to Clinical Model: Autonomous Coding-Agents for Clinician-driven AI Development](https://mlsanity.com/paper/2604.17110/) (2026) - [PEFT-MedSAM: Efficient Fine-Tuning of Medical Foundation Models for Explainable Skin Lesion Segmentation](https://mlsanity.com/paper/2606.18707/) (2026) ## Abstract (arXiv) This paper contributes with a pragmatic evaluation framework for explainable Machine Learning (ML) models for clinical decision support. The study revealed a more nuanced role for ML explanation models, when these are pragmatically embedded in the clinical context. Despite the general positive attitude of healthcare professionals (HCPs) towards explanations as a safety and trust mechanism, for a significant set of participants there were negative effects associated with confirmation bias, accentuating model over-reliance and increased effort to interact with the model. Also, contradicting one of its main intended functions, standard explanatory models showed limited ability to support a critical understanding of the limitations of the model. However, we found new significant positive effects which repositions the role of explanations within a clinical context: these include reduction of automation bias, addressing ambiguous clinical cases (cases where HCPs were not certain about their decision) and support of less experienced HCPs in the acquisition of new domain knowledge.