# What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use

arXiv: https://arxiv.org/abs/1905.05134 · HTML: https://mlsanity.com/paper/1905.05134/

## Key facts

- Cited by 135 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2019.

## Built on by (most cited first, 30 of 135)

- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond](https://mlsanity.com/paper/2102.01998/) (2021)
- [Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs](https://mlsanity.com/paper/2101.09824/) (2021)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Domain Adaptation for Time Series Under Feature and Label Shifts](https://mlsanity.com/paper/2302.03133/) (2023)
- [Concept Activation Regions: A Generalized Framework For Concept-Based Explanations](https://mlsanity.com/paper/2209.11222/) (2022)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [A Survey on Graph-Based Deep Learning for Computational Histopathology](https://mlsanity.com/paper/2107.00272/) (2021)
- [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025)
- [Explainable Deep Learning Methods in Medical Image Classification: A Survey](https://mlsanity.com/paper/2205.04766/) (2022)
- [The Role of Explainability in Assuring Safety of Machine Learning in Healthcare](https://mlsanity.com/paper/2109.00520/) (2021)
- [A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?](https://mlsanity.com/paper/2304.04780/) (2023)
- [Best of both worlds: local and global explanations with human-understandable concepts](https://mlsanity.com/paper/2106.08641/) (2021)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma](https://mlsanity.com/paper/2303.12806/) (2023)
- [Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews](https://mlsanity.com/paper/2305.11828/) (2023)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View](https://mlsanity.com/paper/2112.02625/) (2021)
- [Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficient](https://mlsanity.com/paper/2202.05302/) (2022)
- [Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysis](https://mlsanity.com/paper/2203.16141/) (2022)
- [VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Models](https://mlsanity.com/paper/2108.02550/) (2021)
- [RELAX: Representation Learning Explainability](https://mlsanity.com/paper/2112.10161/) (2021)
- [One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images](https://mlsanity.com/paper/2107.05047/) (2021)
- [Saliency Cards: A Framework to Characterize and Compare Saliency Methods](https://mlsanity.com/paper/2206.02958/) (2022)
- [Intuitively Assessing ML Model Reliability through Example-Based Explanations and Editing Model Inputs](https://mlsanity.com/paper/2102.08540/) (2021)
- [Meta-learning in healthcare: A survey](https://mlsanity.com/paper/2308.02877/) (2023)
- [On the amplification of security and privacy risks by post-hoc explanations in machine learning models](https://mlsanity.com/paper/2206.14004/) (2022)
- [Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes](https://mlsanity.com/paper/2302.05752/) (2023)
- [Learning to Predict with Supporting Evidence: Applications to Clinical Risk Prediction](https://mlsanity.com/paper/2103.02768/) (2021)

## Abstract (arXiv)

Translating machine learning (ML) models effectively to clinical practice requires establishing clinicians' trust. Explainability, or the ability of an ML model to justify its outcomes and assist clinicians in rationalizing the model prediction, has been generally understood to be critical to establishing trust. However, the field suffers from the lack of concrete definitions for usable explanations in different settings. To identify specific aspects of explainability that may catalyze building trust in ML models, we surveyed clinicians from two distinct acute care specialties (Intenstive Care Unit and Emergency Department). We use their feedback to characterize when explainability helps to improve clinicians' trust in ML models. We further identify the classes of explanations that clinicians identified as most relevant and crucial for effective translation to clinical practice. Finally, we discern concrete metrics for rigorous evaluation of clinical explainability methods. By integrating perceptions of explainability between clinicians and ML researchers we hope to facilitate the endorsement and broader adoption and sustained use of ML systems in healthcare.