# Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Review

arXiv: https://arxiv.org/abs/2112.12596 · HTML: https://mlsanity.com/paper/2112.12596/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 136 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 30 of 42)

- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [Explainable artificial intelligence (XAI) in deep learning-based medical image analysis](https://mlsanity.com/paper/2107.10912/) (2021)
- [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021)
- [Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation](https://mlsanity.com/paper/1705.08302/) (2017)
- [Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making](https://mlsanity.com/paper/1902.02960/) (2019)
- [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019)
- [Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved](https://mlsanity.com/paper/1811.11154/) (2018)
- [The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies](https://mlsanity.com/paper/2007.15911/) (2020)
- [Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination](https://mlsanity.com/paper/1408.6491/) (2014)
- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [Understanding artificial intelligence ethics and safety](https://mlsanity.com/paper/1906.05684/) (2019)
- [Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs](https://mlsanity.com/paper/2101.09824/) (2021)
- [CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysis](https://mlsanity.com/paper/2001.05149/) (2020)
- [Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens](https://mlsanity.com/paper/2102.00593/) (2021)
- [The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems](https://mlsanity.com/paper/2008.09100/) (2020)
- [Act Like a Radiologist: Towards Reliable Multi-view Correspondence Reasoning for Mammogram Mass Detection](https://mlsanity.com/paper/2105.10160/) (2021)
- [SAUNet: Shape Attentive U-Net for Interpretable Medical Image Segmentation](https://mlsanity.com/paper/2001.07645/) (2020)
- [Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images: Learning from Radiology Reports and Label Ontology](https://mlsanity.com/paper/1904.04661/) (2019)
- [Learning Interpretable Anatomical Features Through Deep Generative Models: Application to Cardiac Remodeling](https://mlsanity.com/paper/1807.06843/) (2018)
- [Interpretable Deep Models for Cardiac Resynchronisation Therapy Response Prediction](https://mlsanity.com/paper/2006.13811/) (2020)
- [ExplAIn: Explanatory Artificial Intelligence for Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2008.05731/) (2020)
- [Respond-CAM: Analyzing Deep Models for 3D Imaging Data by Visualizations](https://mlsanity.com/paper/1806.00102/) (2018)
- [Brain Biomarker Interpretation in ASD Using Deep Learning and fMRI](https://mlsanity.com/paper/1808.08296/) (2018)
- [Deep Hiearchical Multi-Label Classification Applied to Chest X-Ray Abnormality Taxonomies](https://mlsanity.com/paper/2009.05609/) (2020)

## Built on by (most cited first, 30 of 30)

- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets](https://mlsanity.com/paper/2412.01496/) (2024)
- [Human-AI collaboration is not very collaborative yet: A taxonomy of interaction patterns in AI-assisted decision making from a systematic review](https://mlsanity.com/paper/2310.19778/) (2023)
- [CytoSAE: Interpretable Cell Embeddings for Hematology](https://mlsanity.com/paper/2507.12464/) (2025)
- [Multi-step retrieval and reasoning improves radiology question answering with large language models](https://mlsanity.com/paper/2508.00743/) (2025)
- [DWARF: Disease-weighted network for attention map refinement](https://mlsanity.com/paper/2406.17032/) (2024)
- [A clinically motivated self-supervised approach for content-based image retrieval of CT liver images](https://mlsanity.com/paper/2207.04812/) (2022)
- [Leveraging Brain Modularity Prior for Interpretable Representation Learning of fMRI](https://mlsanity.com/paper/2306.14080/) (2023)
- [Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation](https://mlsanity.com/paper/2308.07528/) (2023)
- [Gaze-Assisted Medical Image Segmentation](https://mlsanity.com/paper/2410.17920/) (2024)
- [Trustworthy Multi-phase Liver Tumor Segmentation via Evidence-based Uncertainty](https://mlsanity.com/paper/2305.05344/) (2023)
- [Interpretable Clinical Classification with Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2509.16750/) (2025)
- [Explainable AI for clinical risk prediction: a survey of concepts, methods, and modalities](https://mlsanity.com/paper/2308.08407/) (2023)
- [Leveraging CAM Algorithms for Explaining Medical Semantic Segmentation](https://mlsanity.com/paper/2409.20287/) (2024)
- [Derivative-Free Diffusion Manifold-Constrained Gradient for Unified XAI](https://mlsanity.com/paper/2411.15265/) (2024)
- [h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective](https://mlsanity.com/paper/2506.17968/) (2025)
- [Radiomic fingerprints for knee MR images assessment](https://mlsanity.com/paper/2506.20306/) (2025)
- [Benchmarking bias: Expanding clinical AI model card to incorporate bias reporting of social and non-social factors](https://mlsanity.com/paper/2311.12560/) (2023)
- [Rad4XCNN: a new agnostic method for post-hoc global explanation of CNN-derived features by means of radiomics](https://mlsanity.com/paper/2405.02334/) (2024)
- [Applications of Explainable artificial intelligence in Earth system science](https://mlsanity.com/paper/2406.11882/) (2024)
- [Explainable AI Enhances Glaucoma Referrals, Yet the Human-AI Team Still Falls Short of the AI Alone](https://mlsanity.com/paper/2407.11974/) (2024)
- [What is Implementation Science; and Why It Matters for Bridging the Artificial Intelligence Innovation-to-Application Gap in Medical Imaging](https://mlsanity.com/paper/2510.13006/) (2025)
- [Methodological Explainability Evaluation of an Interpretable Deep Learning Model for Post-Hepatectomy Liver Failure Prediction Incorporating Counterfactual Explanations and Layerwise Relevance Propagation: A Prospective In Silico Trial](https://mlsanity.com/paper/2408.03771/) (2024)
- [An Intrinsically Explainable Approach to Detecting Vertebral Compression Fractures in CT Scans via Neurosymbolic Modeling](https://mlsanity.com/paper/2412.17258/) (2024)
- [Enhancing Cancer Diagnosis with Explainable & Trustworthy Deep Learning Models](https://mlsanity.com/paper/2412.17527/) (2024)
- [Parameter-Free Bio-Inspired Channel Attention for Enhanced Cardiac MRI Reconstruction](https://mlsanity.com/paper/2505.23872/) (2025)
- [Learning to Seek Evidence: A Verifiable Reasoning Agent with Causal Faithfulness Analysis](https://mlsanity.com/paper/2511.01425/) (2025)
- [Interpretable and backpropagation-free Green Learning for efficient multi-task echocardiographic segmentation and classification](https://mlsanity.com/paper/2601.19743/) (2026)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)

## Abstract (arXiv)

Transparency in Machine Learning (ML), attempts to reveal the working mechanisms of complex models. Transparent ML promises to advance human factors engineering goals of human-centered AI in the target users. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e. a relationship between algorithm and user; as a result, iterative prototyping and evaluation with users is critical to attaining adequate solutions that afford transparency. However, following human-centered design principles in healthcare and medical image analysis is challenging due to the limited availability of and access to end users. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature. Our review reveals multiple severe shortcomings in the design and validation of transparent ML for medical image analysis applications. We find that most studies to date approach transparency as a property of the model itself, similar to task performance, without considering end users during neither development nor evaluation. Additionally, the lack of user research, and the sporadic validation of transparency claims put contemporary research on transparent ML for medical image analysis at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research while acknowledging the challenges of human-centered design in healthcare, we introduce the INTRPRT guideline, a systematic design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests formative user research as the first step of transparent model design to understand user needs and domain requirements. Following this process produces evidence to support design choices, and ultimately, increases the likelihood that the algorithms afford transparency.