# Inherently Interpretable Multi-Label Classification Using Class-Specific Counterfactuals

arXiv: https://arxiv.org/abs/2303.00500 · HTML: https://mlsanity.com/paper/2303.00500/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images](https://mlsanity.com/paper/1808.08114/) (2018)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [Explanation by Progressive Exaggeration](https://mlsanity.com/paper/1911.00483/) (2019)
- [Convolutional Dynamic Alignment Networks for Interpretable Classifications](https://mlsanity.com/paper/2104.00032/) (2021)
- [Measurable Counterfactual Local Explanations for Any Classifier](https://mlsanity.com/paper/1908.03020/) (2019)
- [Visual Feature Attribution using Wasserstein GANs](https://mlsanity.com/paper/1711.08998/) (2017)
- [Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling](https://mlsanity.com/paper/2005.14480/) (2020)
- [Using StyleGAN for Visual Interpretability of Deep Learning Models on Medical Images](https://mlsanity.com/paper/2101.07563/) (2021)
- [CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Nets](https://mlsanity.com/paper/2009.05199/) (2020)
- [xGEMs: Generating Examplars to Explain Black-Box Models](https://mlsanity.com/paper/1806.08867/) (2018)
- [ICAM: Interpretable Classification via Disentangled Representations and Feature Attribution Mapping](https://mlsanity.com/paper/2006.08287/) (2020)
- [Attri-VAE: attribute-based interpretable representations of medical images with variational autoencoders](https://mlsanity.com/paper/2203.10417/) (2022)

## Built on by (most cited first, 10 of 10)

- [Right for the Wrong Reason: Can Interpretable ML Techniques Detect Spurious Correlations?](https://mlsanity.com/paper/2307.12344/) (2023)
- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Counterfactual contrastive learning: robust representations via causal image synthesis](https://mlsanity.com/paper/2403.09605/) (2024)
- [A Framework for Interpretability in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2310.01685/) (2023)
- [Conditional Diffusion Models are Medical Image Classifiers that Provide Explainability and Uncertainty for Free](https://mlsanity.com/paper/2502.03687/) (2025)
- [Label-free Concept Based Multiple Instance Learning for Gigapixel Histopathology](https://mlsanity.com/paper/2501.02922/) (2025)
- [DeCoDEx: Confounder Detector Guidance for Improved Diffusion-based Counterfactual Explanations](https://mlsanity.com/paper/2405.09288/) (2024)
- [Generating Realistic Counterfactuals for Retinal Fundus and OCT Images using Diffusion Models](https://mlsanity.com/paper/2311.11629/) (2023)
- [Is visual explanation with Grad-CAM more reliable for deeper neural networks? a case study with automatic pneumothorax diagnosis](https://mlsanity.com/paper/2308.15172/) (2023)
- [DifCluE: Generating Counterfactual Explanations with Diffusion Autoencoders and modal clustering](https://mlsanity.com/paper/2502.11509/) (2025)

## Abstract (arXiv)

Interpretability is essential for machine learning algorithms in high-stakes application fields such as medical image analysis. However, high-performing black-box neural networks do not provide explanations for their predictions, which can lead to mistrust and suboptimal human-ML collaboration. Post-hoc explanation techniques, which are widely used in practice, have been shown to suffer from severe conceptual problems. Furthermore, as we show in this paper, current explanation techniques do not perform adequately in the multi-label scenario, in which multiple medical findings may co-occur in a single image. We propose Attri-Net, an inherently interpretable model for multi-label classification. Attri-Net is a powerful classifier that provides transparent, trustworthy, and human-understandable explanations. The model first generates class-specific attribution maps based on counterfactuals to identify which image regions correspond to certain medical findings. Then a simple logistic regression classifier is used to make predictions based solely on these attribution maps. We compare Attri-Net to five post-hoc explanation techniques and one inherently interpretable classifier on three chest X-ray datasets. We find that Attri-Net produces high-quality multi-label explanations consistent with clinical knowledge and has comparable classification performance to state-of-the-art classification models.