# Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare

arXiv: https://arxiv.org/abs/2211.16444 · HTML: https://mlsanity.com/paper/2211.16444/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 143 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 19 of 19)

- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Techniques for Interpretable Machine Learning](https://mlsanity.com/paper/1808.00033/) (2018)
- [Explaining Explanations in AI](https://mlsanity.com/paper/1811.01439/) (2018)
- [Expanding Explainability: Towards Social Transparency in AI systems](https://mlsanity.com/paper/2101.04719/) (2021)
- [Explainable deep learning models in medical image analysis](https://mlsanity.com/paper/2005.13799/) (2020)
- [Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making](https://mlsanity.com/paper/1902.02960/) (2019)
- [The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies](https://mlsanity.com/paper/2007.15911/) (2020)
- [Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond](https://mlsanity.com/paper/2102.01998/) (2021)
- [Understanding artificial intelligence ethics and safety](https://mlsanity.com/paper/1906.05684/) (2019)
- [Trustworthy AI](https://mlsanity.com/paper/2011.02272/) (2020)
- [Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes](https://mlsanity.com/paper/1904.09612/) (2019)
- [Detection of data drift and outliers affecting machine learning model performance over time](https://mlsanity.com/paper/2012.09258/) (2020)
- [Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology](https://mlsanity.com/paper/1806.05083/) (2018)
- [Survey of XAI in digital pathology](https://mlsanity.com/paper/2008.06353/) (2020)
- [Open the Black Box Data-Driven Explanation of Black Box Decision Systems](https://mlsanity.com/paper/1806.09936/) (2018)
- [Trust, Professional Vision and Diagnostic Work](https://mlsanity.com/paper/2205.11614/) (2022)

## Built on by (most cited first, 9 of 9)

- [HealthGenie: Empowering Users with Healthy Dietary Guidance through Knowledge Graph and Large Language Models](https://mlsanity.com/paper/2504.14594/) (2025)
- [Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology](https://mlsanity.com/paper/2405.05299/) (2024)
- [A Design Framework for operationalizing Trustworthy Artificial Intelligence in Healthcare: Requirements, Tradeoffs and Challenges for its Clinical Adoption](https://mlsanity.com/paper/2504.19179/) (2025)
- [Visual-Conversational Interface for Evidence-Based Explanation of Diabetes Risk Prediction](https://mlsanity.com/paper/2507.02920/) (2025)
- [Blockchain-Enabled Explainable AI for Trusted Healthcare Systems](https://mlsanity.com/paper/2509.14987/) (2025)
- ["It depends": Configuring AI to Improve Clinical Usefulness Across Contexts](https://mlsanity.com/paper/2407.11978/) (2024)
- [Trustworthy AI: UK Air Traffic Control Revisited](https://mlsanity.com/paper/2507.21169/) (2025)
- [Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AI](https://mlsanity.com/paper/2603.20595/) (2026)
- [CoPlan: A Trustworthy Co-Intelligence Interface for Care Planning through Role-Based Contestable Argument Graphs](https://mlsanity.com/paper/2608.05107/) (2026)

## Abstract (arXiv)

The need for AI systems to provide explanations for their behaviour is now widely recognised as key to their adoption. In this paper, we examine the problem of trustworthy AI and explore what delivering this means in practice, with a focus on healthcare applications. Work in this area typically treats trustworthy AI as a problem of Human-Computer Interaction involving the individual user and an AI system. However, we argue here that this overlooks the important part played by organisational accountability in how people reason about and trust AI in socio-technical settings. To illustrate the importance of organisational accountability, we present findings from ethnographic studies of breast cancer screening and cancer treatment planning in multidisciplinary team meetings to show how participants made themselves accountable both to each other and to the organisations of which they are members. We use these findings to enrich existing understandings of the requirements for trustworthy AI and to outline some candidate solutions to the problems of making AI accountable both to individual users and organisationally. We conclude by outlining the implications of this for future work on the development of trustworthy AI, including ways in which our proposed solutions may be re-used in different application settings.