# Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making

arXiv: https://arxiv.org/abs/1902.02960 · HTML: https://mlsanity.com/paper/1902.02960/

## Key facts

- Cited by 109 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2019.

## Built on by (most cited first, 30 of 109)

- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty](https://mlsanity.com/paper/2401.06730/) (2024)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making](https://mlsanity.com/paper/2101.05303/) (2021)
- [Designing a Dashboard for Transparency and Control of Conversational AI](https://mlsanity.com/paper/2406.07882/) (2024)
- [Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making](https://mlsanity.com/paper/2403.16812/) (2024)
- [DISSECT: Disentangled Simultaneous Explanations via Concept Traversals](https://mlsanity.com/paper/2105.15164/) (2021)
- [Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis](https://mlsanity.com/paper/2202.10553/) (2022)
- [Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm](https://mlsanity.com/paper/2207.02942/) (2022)
- [Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging](https://mlsanity.com/paper/2205.09696/) (2022)
- [Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Review](https://mlsanity.com/paper/2112.12596/) (2021)
- [Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis](https://mlsanity.com/paper/2309.12368/) (2023)
- [Selective Explanations: Leveraging Human Input to Align Explainable AI](https://mlsanity.com/paper/2301.09656/) (2023)
- ["Brilliant AI Doctor" in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment](https://mlsanity.com/paper/2101.01524/) (2021)
- [Towards Relatable Explainable AI with the Perceptual Process](https://mlsanity.com/paper/2112.14005/) (2021)
- [Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations](https://mlsanity.com/paper/2112.09669/) (2021)
- [Probabilistically robust conformal prediction](https://mlsanity.com/paper/2307.16360/) (2023)
- [Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty](https://mlsanity.com/paper/2412.06771/) (2024)
- [Human-Centered Concept Explanations for Neural Networks](https://mlsanity.com/paper/2202.12451/) (2022)
- [Best of both worlds: local and global explanations with human-understandable concepts](https://mlsanity.com/paper/2106.08641/) (2021)
- [Interactive AI Alignment: Specification, Process, and Evaluation Alignment](https://mlsanity.com/paper/2311.00710/) (2023)
- [How Much Automation Does a Data Scientist Want?](https://mlsanity.com/paper/2101.03970/) (2021)
- [Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorations](https://mlsanity.com/paper/2302.10671/) (2023)
- [Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making](https://mlsanity.com/paper/2308.04375/) (2023)

## Abstract (arXiv)

Machine learning (ML) is increasingly being used in image retrieval systems for medical decision making. One application of ML is to retrieve visually similar medical images from past patients (e.g. tissue from biopsies) to reference when making a medical decision with a new patient. However, no algorithm can perfectly capture an expert's ideal notion of similarity for every case: an image that is algorithmically determined to be similar may not be medically relevant to a doctor's specific diagnostic needs. In this paper, we identified the needs of pathologists when searching for similar images retrieved using a deep learning algorithm, and developed tools that empower users to cope with the search algorithm on-the-fly, communicating what types of similarity are most important at different moments in time. In two evaluations with pathologists, we found that these refinement tools increased the diagnostic utility of images found and increased user trust in the algorithm. The tools were preferred over a traditional interface, without a loss in diagnostic accuracy. We also observed that users adopted new strategies when using refinement tools, re-purposing them to test and understand the underlying algorithm and to disambiguate ML errors from their own errors. Taken together, these findings inform future human-ML collaborative systems for expert decision-making.