Before the Clinic: Transparent and Operable Design Principles for Healthcare AI
Not on the map. Why?
Abstract (arXiv)
The translation of artificial intelligence (AI) systems into clinical practice requires bridging fundamental gaps between explainable AI theory, clinician expectations, and governance requirements. While conceptual frameworks define what constitutes explainable AI (XAI) and qualitative studies identify clinician needs, little practical guidance exists for development teams to prepare AI systems prior to clinical evaluation. We propose two foundational design principles, Transparent Design and Operable Design, that operationalize pre-clinical technical requirements for healthcare AI. Transparent Design encompasses interpretability and understandability artifacts that enable case-level reasoning and system traceability. Operable Design encompasses calibration, uncertainty, and robustness to ensure reliable, predictable system behavior under real-world conditions. We ground these principles in established XAI frameworks, map them to documented clinician needs, and demonstrate their alignment with emerging governance requirements. This pre-clinical playbook provides actionable guidance for development teams, accelerates the path to clinical evaluation, and establishes a shared vocabulary bridging AI researchers, healthcare practitioners, and regulatory stakeholders. By explicitly scoping what can be built and verified before clinical deployment, we aim to reduce friction in clinical AI translation while remaining cautious about what constitutes validated, deployed explainability.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 54 references that are in the map. First posted on arXiv 31 Oct 2025.
Builds on
19 of 54 references.
References · 19
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2017Axiomatic Attribution for Deep Networkscited by 3,350
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
- 2018Datasheets for Datasetscited by 1,371
- 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
- 2018Sanity Checks for Saliency Mapscited by 1,149
- 2019Attention is not Explanationcited by 916
- 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
- 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
7 more
- 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
- 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 440
- 2016RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanismcited by 344
- 2018On the Robustness of Interpretability Methodscited by 310
- 2019What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Usecited by 134
- 2020Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodologycited by 41
- 2019"The Human Body is a Black Box": Supporting Clinical Decision-Making with Deep Learningcited by 39
Built on by
Citing papers per year: 2025: 1.
1 paper.