# Interpreting Medical Image Classifiers by Optimization Based Counterfactual Impact Analysis

arXiv: https://arxiv.org/abs/2004.01610 · HTML: https://mlsanity.com/paper/2004.01610/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys](https://mlsanity.com/paper/2310.10765/) (2023)
- [Challenges for machine learning in clinical translation of big data imaging studies](https://mlsanity.com/paper/2107.05630/) (2021)
- [Multi-task fusion for improving mammography screening data classification](https://mlsanity.com/paper/2112.01320/) (2021)
- [Leveraging Conditional Generative Models in a General Explanation Framework of Classifier Decisions](https://mlsanity.com/paper/2106.10947/) (2021)
- [Leveraging Causal Reasoning Method for Explaining Medical Image Segmentation Models](https://mlsanity.com/paper/2602.20511/) (2026)

## Abstract (arXiv)

Clinical applicability of automated decision support systems depends on a robust, well-understood classification interpretation. Artificial neural networks while achieving class-leading scores fall short in this regard. Therefore, numerous approaches have been proposed that map a salient region of an image to a diagnostic classification. Utilizing heuristic methodology, like blurring and noise, they tend to produce diffuse, sometimes misleading results, hindering their general adoption. In this work we overcome these issues by presenting a model agnostic saliency mapping framework tailored to medical imaging. We replace heuristic techniques with a strong neighborhood conditioned inpainting approach, which avoids anatomically implausible artefacts. We formulate saliency attribution as a map-quality optimization task, enforcing constrained and focused attributions. Experiments on public mammography data show quantitatively and qualitatively more precise localization and clearer conveying results than existing state-of-the-art methods.