# Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework arXiv: https://arxiv.org/abs/2311.11055 ยท HTML: https://mlsanity.com/paper/2311.11055/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 147 references that are in the map. - First posted on arXiv 18 Nov 2023. ## Builds on (most cited first, 25 of 25) - [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017) - [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018) - [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017) - [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018) - [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019) - [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019) - [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019) - [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016) - [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019) - [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018) - [Drug discovery with explainable artificial intelligence](https://mlsanity.com/paper/2007.00523/) (2020) - [Artificial Intelligence: the global landscape of ethics guidelines](https://mlsanity.com/paper/1906.11668/) (2019) - [Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond](https://mlsanity.com/paper/2102.01998/) (2021) - [IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography](https://mlsanity.com/paper/2103.12308/) (2021) - [Towards Interpretable Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2304.03347/) (2023) - [TED: Teaching AI to Explain its Decisions](https://mlsanity.com/paper/1811.04896/) (2018) - [EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case](https://mlsanity.com/paper/2104.11914/) (2021) - [Explainable AI for clinical and remote health applications: a survey on tabular and time series data](https://mlsanity.com/paper/2209.06528/) (2022) - [Sibyl: Understanding and Addressing the Usability Challenges of Machine Learning In High-Stakes Decision Making](https://mlsanity.com/paper/2103.02071/) (2021) - [Deep Weighted Averaging Classifiers](https://mlsanity.com/paper/1811.02579/) (2018) - [Missing Values and Imputation in Healthcare Data: Can Interpretable Machine Learning Help?](https://mlsanity.com/paper/2304.11749/) (2023) - [A novel interpretable machine learning system to generate clinical risk scores: An application for predicting early mortality or unplanned readmission in a retrospective cohort study](https://mlsanity.com/paper/2201.03291/) (2022) - [Parallel Coordinate Order for High-Dimensional Data](https://mlsanity.com/paper/1905.10035/) (2019) - [A New Approach for Interpretability and Reliability in Clinical Risk Prediction: Acute Coronary Syndrome Scenario](https://mlsanity.com/paper/2110.08331/) (2021) - [AI Framework for Early Diagnosis of Coronary Artery Disease: An Integration of Borderline SMOTE, Autoencoders and Convolutional Neural Networks Approach](https://mlsanity.com/paper/2308.15339/) (2023) ## Built on by (most cited first, 11 of 11) - [Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis](https://mlsanity.com/paper/2403.16303/) (2024) - [LLM-Agent-UMF: LLM-based Agent Unified Modeling Framework for Seamless Design of Multi Active/Passive Core-Agent Architectures](https://mlsanity.com/paper/2409.11393/) (2024) - [A Design Framework for operationalizing Trustworthy Artificial Intelligence in Healthcare: Requirements, Tradeoffs and Challenges for its Clinical Adoption](https://mlsanity.com/paper/2504.19179/) (2025) - [Ontology- and LLM-based Data Harmonization for Federated Learning in Healthcare](https://mlsanity.com/paper/2505.20020/) (2025) - [Making Sense of AI Limitations: How Individual Perceptions Shape Organizational Readiness for AI Adoption](https://mlsanity.com/paper/2502.15870/) (2025) - [Interpretable Multimodal Learning for Tumor Protein-Metal Binding: Progress, Challenges, and Perspectives](https://mlsanity.com/paper/2504.03847/) (2025) - [Learning Label-Efficient Interpretable Medical Image Diagnosis via Semi-supervised Hypergraph Concept Bottleneck Model](https://mlsanity.com/paper/2606.01698/) (2026) - [Bridging AI Innovation and Healthcare Needs: Lessons Learned from Incorporating Modern NLP at The BC Cancer Registry](https://mlsanity.com/paper/2508.09991/) (2025) - [Self-Explainability in Self-Adaptive and Self-Organising Systems: Status and Research Directions](https://mlsanity.com/paper/2606.09568/) (2026) - [TraMP-LLaMA: Generative Interpretability with Decoupled Instruction Tuning for Facial Expression Quality Assessment](https://mlsanity.com/paper/2606.26942/) (2026) - [SIFTING: A Novel LLM-Based Framework for Structured and Transparent Information Extraction from Clinical Free-Text Reports, with Application to Tumor Staging in Lung Cancer](https://mlsanity.com/paper/2609.07185/) (2026) ## Abstract (arXiv) This paper explores the significant impact of AI-based medical devices, including wearables, telemedicine, large language models, and digital twins, on clinical decision support systems. It emphasizes the importance of producing outcomes that are not only accurate but also interpretable and understandable to clinicians, addressing the risk that lack of interpretability poses in terms of mistrust and reluctance to adopt these technologies in healthcare. The paper reviews interpretable AI processes, methods, applications, and the challenges of implementation in healthcare, focusing on quality control to facilitate responsible communication between AI systems and clinicians. It breaks down the interpretability process into data pre-processing, model selection, and post-processing, aiming to foster a comprehensive understanding of the crucial role of a robust interpretability approach in healthcare and to guide future research in this area. with insights for creating responsible clinician-AI tools for healthcare, as well as to offer a deeper understanding of the challenges they might face. Our research questions, eligibility criteria and primary goals were identified using Preferred Reporting Items for Systematic reviews and Meta-Analyses guideline and PICO method; PubMed, Scopus and Web of Science databases were systematically searched using sensitive and specific search strings. In the end, 52 publications were selected for data extraction which included 8 existing reviews and 44 related experimental studies. The paper offers general concepts of interpretable AI in healthcare and discuss three-levels interpretability process. Additionally, it provides a comprehensive discussion of evaluating robust interpretability AI in healthcare. Moreover, this survey introduces a step-by-step roadmap for implementing responsible AI in healthcare.