# High Fidelity Image Counterfactuals with Probabilistic Causal Models

arXiv: https://arxiv.org/abs/2306.15764 · HTML: https://mlsanity.com/paper/2306.15764/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 96 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 30 of 64)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)

## Built on by (most cited first, 30 of 53)

- [Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation](https://mlsanity.com/paper/2312.14223/) (2023)
- [From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling](https://mlsanity.com/paper/2310.11011/) (2023)
- [Benchmarking Counterfactual Image Generation](https://mlsanity.com/paper/2403.20287/) (2024)
- [Counterfactual Image Editing](https://mlsanity.com/paper/2403.09683/) (2024)
- [PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion](https://mlsanity.com/paper/2503.00196/) (2025)
- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Diffusion-based Iterative Counterfactual Explanations for Fetal Ultrasound Image Quality Assessment](https://mlsanity.com/paper/2403.08700/) (2024)
- [Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis](https://mlsanity.com/paper/2412.20651/) (2024)
- [Mitigating attribute amplification in counterfactual image generation](https://mlsanity.com/paper/2403.09422/) (2024)
- [No Fair Lunch: A Causal Perspective on Dataset Bias in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2307.16526/) (2023)
- [Counterfactual contrastive learning: robust representations via causal image synthesis](https://mlsanity.com/paper/2403.09605/) (2024)
- [Counterfactual Generative Modeling with Variational Causal Inference](https://mlsanity.com/paper/2410.12730/) (2024)
- [Diffusion Counterfactual Generation with Semantic Abduction](https://mlsanity.com/paper/2506.07883/) (2025)
- [CF-Seg: Counterfactuals meet Segmentation](https://mlsanity.com/paper/2506.16213/) (2025)
- [Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference](https://mlsanity.com/paper/2401.01426/) (2024)
- [Latent Causal Modeling for 3D Brain MRI Counterfactuals](https://mlsanity.com/paper/2409.05585/) (2024)
- [Semi-Supervised Learning for Deep Causal Generative Models](https://mlsanity.com/paper/2403.18717/) (2024)
- [Causally Steered Diffusion for Automated Video Counterfactual Generation](https://mlsanity.com/paper/2506.14404/) (2025)
- [Counterfactual Identifiability via Dynamic Optimal Transport](https://mlsanity.com/paper/2510.08294/) (2025)
- [Overcoming Statistical Bias in Action-Controllable World Models](https://mlsanity.com/paper/2608.04653/) (2026)
- [Conditional Generative Models are Sufficient to Sample from Any Causal Effect Estimand](https://mlsanity.com/paper/2402.07419/) (2024)
- [Synthetic Data for Robust Stroke Segmentation](https://mlsanity.com/paper/2404.01946/) (2024)
- [Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation](https://mlsanity.com/paper/2509.24798/) (2025)
- [MACAW: A Causal Generative Model for Medical Imaging](https://mlsanity.com/paper/2412.02900/) (2024)
- [Causal Disentanglement for Robust Long-tail Medical Image Generation](https://mlsanity.com/paper/2504.14450/) (2025)
- [Flow Stochastic Segmentation Networks](https://mlsanity.com/paper/2507.18838/) (2025)
- [SSL Framework for Causal Inconsistency between Structures and Representations](https://mlsanity.com/paper/2310.18634/) (2023)
- [Evaluation of 3D Counterfactual Brain MRI Generation](https://mlsanity.com/paper/2508.02880/) (2025)
- [Segmentor-Guided Counterfactual Fine-Tuning for Locally Coherent and Targeted Image Synthesis](https://mlsanity.com/paper/2509.24913/) (2025)
- [Probabilistic Modelling is Sufficient for Causal Inference](https://mlsanity.com/paper/2512.23408/) (2025)

## Abstract (arXiv)

We present a general causal generative modelling framework for accurate estimation of high fidelity image counterfactuals with deep structural causal models. Estimation of interventional and counterfactual queries for high-dimensional structured variables, such as images, remains a challenging task. We leverage ideas from causal mediation analysis and advances in generative modelling to design new deep causal mechanisms for structured variables in causal models. Our experiments demonstrate that our proposed mechanisms are capable of accurate abduction and estimation of direct, indirect and total effects as measured by axiomatic soundness of counterfactuals.