# Evaluating and Mitigating Bias in Image Classifiers: A Causal Perspective Using Counterfactuals

arXiv: https://arxiv.org/abs/2009.08270 · HTML: https://mlsanity.com/paper/2009.08270/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 30 of 56)

- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [Fast Model Debias with Machine Unlearning](https://mlsanity.com/paper/2310.12560/) (2023)
- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [Probing Classifiers are Unreliable for Concept Removal and Detection](https://mlsanity.com/paper/2207.04153/) (2022)
- [Measuring axiomatic soundness of counterfactual image models](https://mlsanity.com/paper/2303.01274/) (2023)
- [UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models](https://mlsanity.com/paper/2110.15499/) (2021)
- [Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey](https://mlsanity.com/paper/2211.05617/) (2022)
- [Benchmarking Counterfactual Image Generation](https://mlsanity.com/paper/2403.20287/) (2024)
- [Counterfactual Image Editing](https://mlsanity.com/paper/2403.09683/) (2024)
- [Counterfactual (Non-)identifiability of Learned Structural Causal Models](https://mlsanity.com/paper/2301.09031/) (2023)
- [Ethical Considerations for Responsible Data Curation](https://mlsanity.com/paper/2302.03629/) (2023)
- [The role of causality in explainable artificial intelligence](https://mlsanity.com/paper/2309.09901/) (2023)
- [Learn-Explain-Reinforce: Counterfactual Reasoning and Its Guidance to Reinforce an Alzheimer's Disease Diagnosis Model](https://mlsanity.com/paper/2108.09451/) (2021)
- [Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Model](https://mlsanity.com/paper/2306.01424/) (2023)
- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Improving Fairness using Vision-Language Driven Image Augmentation](https://mlsanity.com/paper/2311.01573/) (2023)
- [Counterfactual contrastive learning: robust representations via causal image synthesis](https://mlsanity.com/paper/2403.09605/) (2024)
- [Counterfactual Generative Modeling with Variational Causal Inference](https://mlsanity.com/paper/2410.12730/) (2024)
- [Matching Learned Causal Effects of Neural Networks with Domain Priors](https://mlsanity.com/paper/2111.12490/) (2021)
- [Diffusion Counterfactual Generation with Semantic Abduction](https://mlsanity.com/paper/2506.07883/) (2025)
- [Deep Backtracking Counterfactuals for Causally Compliant Explanations](https://mlsanity.com/paper/2310.07665/) (2023)
- [Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference](https://mlsanity.com/paper/2401.01426/) (2024)
- [Distributionally Generative Augmentation for Fair Facial Attribute Classification](https://mlsanity.com/paper/2403.06606/) (2024)
- [Learning Structural Causal Models through Deep Generative Models: Methods, Guarantees, and Challenges](https://mlsanity.com/paper/2405.05025/) (2024)
- [Adversarial Counterfactual Augmentation: Application in Alzheimer's Disease Classification](https://mlsanity.com/paper/2203.07815/) (2022)
- [Causally Steered Diffusion for Automated Video Counterfactual Generation](https://mlsanity.com/paper/2506.14404/) (2025)
- [AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data](https://mlsanity.com/paper/2503.05665/) (2025)
- [Explaining Image Classifiers Using Contrastive Counterfactuals in Generative Latent Spaces](https://mlsanity.com/paper/2206.05257/) (2022)
- [Combining Counterfactuals With Shapley Values To Explain Image Models](https://mlsanity.com/paper/2206.07087/) (2022)
- [Causal Generative Explainers using Counterfactual Inference: A Case Study on the Morpho-MNIST Dataset](https://mlsanity.com/paper/2401.11394/) (2024)

## Abstract (arXiv)

Counterfactual examples for an input -- perturbations that change specific features but not others -- have been shown to be useful for evaluating bias of machine learning models, e.g., against specific demographic groups. However, generating counterfactual examples for images is non-trivial due to the underlying causal structure on the various features of an image. To be meaningful, generated perturbations need to satisfy constraints implied by the causal model. We present a method for generating counterfactuals by incorporating a structural causal model (SCM) in an improved variant of Adversarially Learned Inference (ALI), that generates counterfactuals in accordance with the causal relationships between attributes of an image. Based on the generated counterfactuals, we show how to explain a pre-trained machine learning classifier, evaluate its bias, and mitigate the bias using a counterfactual regularizer. On the Morpho-MNIST dataset, our method generates counterfactuals comparable in quality to prior work on SCM-based counterfactuals (DeepSCM), while on the more complex CelebA dataset our method outperforms DeepSCM in generating high-quality valid counterfactuals. Moreover, generated counterfactuals are indistinguishable from reconstructed images in a human evaluation experiment and we subsequently use them to evaluate the fairness of a standard classifier trained on CelebA data. We show that the classifier is biased w.r.t. skin and hair color, and how counterfactual regularization can remove those biases.