# Right for the Wrong Reason: Can Interpretable ML Techniques Detect Spurious Correlations?

arXiv: https://arxiv.org/abs/2307.12344 · HTML: https://mlsanity.com/paper/2307.12344/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 23 Jul 2023.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [When Explanations Lie: Why Many Modified BP Attributions Fail](https://mlsanity.com/paper/1912.09818/) (2019)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [Explanation by Progressive Exaggeration](https://mlsanity.com/paper/1911.00483/) (2019)
- [Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation](https://mlsanity.com/paper/2212.04629/) (2022)
- [Convolutional Dynamic Alignment Networks for Interpretable Classifications](https://mlsanity.com/paper/2104.00032/) (2021)
- [Measurable Counterfactual Local Explanations for Any Classifier](https://mlsanity.com/paper/1908.03020/) (2019)
- [Inherently Interpretable Multi-Label Classification Using Class-Specific Counterfactuals](https://mlsanity.com/paper/2303.00500/) (2023)

## Built on by (most cited first, 16 of 16)

- [Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation](https://mlsanity.com/paper/2312.14223/) (2023)
- [Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging](https://mlsanity.com/paper/2403.04484/) (2024)
- [A Framework for Interpretability in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2310.01685/) (2023)
- [DeCoDEx: Confounder Detector Guidance for Improved Diffusion-based Counterfactual Explanations](https://mlsanity.com/paper/2405.09288/) (2024)
- [Mask of truth: model sensitivity to unexpected regions of medical images](https://mlsanity.com/paper/2412.04030/) (2024)
- [Subgroup Performance Analysis in Hidden Stratifications](https://mlsanity.com/paper/2503.10382/) (2025)
- [ViG-Bias: Visually Grounded Bias Discovery and Mitigation](https://mlsanity.com/paper/2407.01996/) (2024)
- [Characterizing the Interpretability of Attention Maps in Digital Pathology](https://mlsanity.com/paper/2407.02484/) (2024)
- [Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging](https://mlsanity.com/paper/2407.18792/) (2024)
- [Prototype-Based Multiple Instance Learning for Gigapixel Whole Slide Image Classification](https://mlsanity.com/paper/2503.08384/) (2025)
- [In-hoc Concept Representations to Regularise Deep Learning in Medical Imaging](https://mlsanity.com/paper/2508.13880/) (2025)
- [Attri-Net: A Globally and Locally Inherently Interpretable Model for Multi-Label Classification Using Class-Specific Counterfactuals](https://mlsanity.com/paper/2406.05477/) (2024)
- [MIMM-X: Disentangling Spurious Correlations for Medical Image Analysis](https://mlsanity.com/paper/2511.22990/) (2025)
- [Enhancing Interpretability of Vertebrae Fracture Grading using Human-interpretable Prototypes](https://mlsanity.com/paper/2404.02830/) (2024)
- [Label-free estimation of clinically relevant performance metrics under distribution shifts](https://mlsanity.com/paper/2507.22776/) (2025)
- [Spatially Grounded Concept-Based Image Classification](https://mlsanity.com/paper/2510.04180/) (2025)

## Abstract (arXiv)

While deep neural network models offer unmatched classification performance, they are prone to learning spurious correlations in the data. Such dependencies on confounding information can be difficult to detect using performance metrics if the test data comes from the same distribution as the training data. Interpretable ML methods such as post-hoc explanations or inherently interpretable classifiers promise to identify faulty model reasoning. However, there is mixed evidence whether many of these techniques are actually able to do so. In this paper, we propose a rigorous evaluation strategy to assess an explanation technique's ability to correctly identify spurious correlations. Using this strategy, we evaluate five post-hoc explanation techniques and one inherently interpretable method for their ability to detect three types of artificially added confounders in a chest x-ray diagnosis task. We find that the post-hoc technique SHAP, as well as the inherently interpretable Attri-Net provide the best performance and can be used to reliably identify faulty model behavior.