# Debiasing Counterfactuals In the Presence of Spurious Correlations

arXiv: https://arxiv.org/abs/2308.10984 · HTML: https://mlsanity.com/paper/2308.10984/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Builds on (most cited first, 13 of 13)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Resnet in Resnet: Generalizing Residual Architectures](https://mlsanity.com/paper/1603.08029/) (2016)
- [MEDFAIR: Benchmarking Fairness for Medical Imaging](https://mlsanity.com/paper/2210.01725/) (2022)
- [GANterfactual - Counterfactual Explanations for Medical Non-Experts using Generative Adversarial Learning](https://mlsanity.com/paper/2012.11905/) (2020)
- [CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Nets](https://mlsanity.com/paper/2009.05199/) (2020)
- [An Explainable Machine Learning Model for Early Detection of Parkinson's Disease using LIME on DaTscan Imagery](https://mlsanity.com/paper/2008.00238/) (2020)
- [Evaluating the Fairness of Deep Learning Uncertainty Estimates in Medical Image Analysis](https://mlsanity.com/paper/2303.03242/) (2023)
- [Counterfactual Image Synthesis for Discovery of Personalized Predictive Image Markers](https://mlsanity.com/paper/2208.02311/) (2022)
- [Mitigating Calibration Bias Without Fixed Attribute Grouping for Improved Fairness in Medical Imaging Analysis](https://mlsanity.com/paper/2307.01738/) (2023)

## Built on by (most cited first, 12 of 12)

- [PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion](https://mlsanity.com/paper/2503.00196/) (2025)
- [CF-Seg: Counterfactuals meet Segmentation](https://mlsanity.com/paper/2506.16213/) (2025)
- [DeCoDEx: Confounder Detector Guidance for Improved Diffusion-based Counterfactual Explanations](https://mlsanity.com/paper/2405.09288/) (2024)
- [RL4Med-DDPO: Reinforcement Learning for Controlled Guidance Towards Diverse Medical Image Generation using Vision-Language Foundation Models](https://mlsanity.com/paper/2503.15784/) (2025)
- [Segmentor-Guided Counterfactual Fine-Tuning for Locally Coherent and Targeted Image Synthesis](https://mlsanity.com/paper/2509.24913/) (2025)
- [Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imaging](https://mlsanity.com/paper/2503.23618/) (2025)
- [Understanding Dataset Bias in Medical Imaging: A Case Study on Chest X-rays](https://mlsanity.com/paper/2507.07722/) (2025)
- [MIMM-X: Disentangling Spurious Correlations for Medical Image Analysis](https://mlsanity.com/paper/2511.22990/) (2025)
- [CEPAE: Conditional Entropy-Penalized Autoencoders for Time Series Counterfactuals](https://mlsanity.com/paper/2602.15546/) (2026)
- [Pixel Perfect MegaMed: A Megapixel-Scale Vision-Language Foundation Model for Generating High Resolution Medical Images](https://mlsanity.com/paper/2507.12698/) (2025)
- [Weight Space Correlation Analysis: Quantifying Feature Utilization in Deep Learning Models](https://mlsanity.com/paper/2512.13144/) (2025)
- [Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesis](https://mlsanity.com/paper/2603.21213/) (2026)

## Abstract (arXiv)

Deep learning models can perform well in complex medical imaging classification tasks, even when basing their conclusions on spurious correlations (i.e. confounders), should they be prevalent in the training dataset, rather than on the causal image markers of interest. This would thereby limit their ability to generalize across the population. Explainability based on counterfactual image generation can be used to expose the confounders but does not provide a strategy to mitigate the bias. In this work, we introduce the first end-to-end training framework that integrates both (i) popular debiasing classifiers (e.g. distributionally robust optimization (DRO)) to avoid latching onto the spurious correlations and (ii) counterfactual image generation to unveil generalizable imaging markers of relevance to the task. Additionally, we propose a novel metric, Spurious Correlation Latching Score (SCLS), to quantify the extent of the classifier reliance on the spurious correlation as exposed by the counterfactual images. Through comprehensive experiments on two public datasets (with the simulated and real visual artifacts), we demonstrate that the debiasing method: (i) learns generalizable markers across the population, and (ii) successfully ignores spurious correlations and focuses on the underlying disease pathology.