# Benchmarking Counterfactual Image Generation

arXiv: https://arxiv.org/abs/2403.20287 · HTML: https://mlsanity.com/paper/2403.20287/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 70 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives](https://mlsanity.com/paper/1802.07623/) (2018)
- [Interpretable Counterfactual Explanations Guided by Prototypes](https://mlsanity.com/paper/1907.02584/) (2019)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [EditGAN: High-Precision Semantic Image Editing](https://mlsanity.com/paper/2111.03186/) (2021)
- [CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training](https://mlsanity.com/paper/1709.02023/) (2017)
- [Counterfactual Generative Networks](https://mlsanity.com/paper/2101.06046/) (2021)
- [Causal Machine Learning: A Survey and Open Problems](https://mlsanity.com/paper/2206.15475/) (2022)

## Built on by (most cited first, 22 of 22)

- [Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models](https://mlsanity.com/paper/2404.17735/) (2024)
- [DanceTogether! Identity-Preserving Multi-Person Interactive Video Generation](https://mlsanity.com/paper/2505.18078/) (2025)
- [Counterfactual Generative Modeling with Variational Causal Inference](https://mlsanity.com/paper/2410.12730/) (2024)
- [Diffusion Counterfactual Generation with Semantic Abduction](https://mlsanity.com/paper/2506.07883/) (2025)
- [Counterfactual Explanations via Riemannian Latent Space Traversal](https://mlsanity.com/paper/2411.02259/) (2024)
- [Causally Steered Diffusion for Automated Video Counterfactual Generation](https://mlsanity.com/paper/2506.14404/) (2025)
- [Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation](https://mlsanity.com/paper/2509.24798/) (2025)
- [MACAW: A Causal Generative Model for Medical Imaging](https://mlsanity.com/paper/2412.02900/) (2024)
- [CausalAffect: Causal Discovery for Facial Affective Understanding](https://mlsanity.com/paper/2512.00456/) (2025)
- [Evaluation of 3D Counterfactual Brain MRI Generation](https://mlsanity.com/paper/2508.02880/) (2025)
- [Error-Driven Scene Editing for 3D Grounding in Large Language Models](https://mlsanity.com/paper/2511.14086/) (2025)
- [When the Edit Changes the Patient: Measuring Identity Preservation in Counterfactual Retinal Images](https://mlsanity.com/paper/2608.23024/) (2026)
- [Advancing oncology with federated learning: transcending boundaries in breast, lung, and prostate cancer. A systematic review](https://mlsanity.com/paper/2408.05249/) (2024)
- [Cyclic Counterfactuals under Shift-Scale Interventions](https://mlsanity.com/paper/2510.25005/) (2025)
- [CoDCL: Counterfactual-Inspired Augmentation Contrastive Learning for Temporal Link Prediction in Social Networks](https://mlsanity.com/paper/2601.22427/) (2026)
- [A Causal Framework for Mitigating Data Shifts in Healthcare](https://mlsanity.com/paper/2603.13595/) (2026)
- [Causal Representation Learning on High-Dimensional Data: Benchmarks, Reproducibility, and Evaluation Metrics](https://mlsanity.com/paper/2603.17405/) (2026)
- [How Many Visual Levers Drive Urban Perception? Interventional Counterfactuals via Multiple Localised Edits](https://mlsanity.com/paper/2604.22103/) (2026)
- [Counterfactual Stress Testing for Image Classification Models](https://mlsanity.com/paper/2605.10894/) (2026)
- [Clinically-Grounded Counterfactual Reasoning for Medical Video Diagnosis](https://mlsanity.com/paper/2605.26483/) (2026)
- [MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation](https://mlsanity.com/paper/2609.15647/) (2026)
- [A Data-Interventional Framework for Auditing Privacy and Fairness in Generative Medical Imaging](https://mlsanity.com/paper/2609.26623/) (2026)

## Abstract (arXiv)

Generative AI has revolutionised visual content editing, empowering users to effortlessly modify images and videos. However, not all edits are equal. To perform realistic edits in domains such as natural image or medical imaging, modifications must respect causal relationships inherent to the data generation process. Such image editing falls into the counterfactual image generation regime. Evaluating counterfactual image generation is substantially complex: not only it lacks observable ground truths, but also requires adherence to causal constraints. Although several counterfactual image generation methods and evaluation metrics exist, a comprehensive comparison within a unified setting is lacking. We present a comparison framework to thoroughly benchmark counterfactual image generation methods. We integrate all models that have been used for the task at hand and expand them to novel datasets and causal graphs, demonstrating the superiority of Hierarchical VAEs across most datasets and metrics. Our framework is implemented in a user-friendly Python package that can be extended to incorporate additional SCMs, causal methods, generative models, and datasets for the community to build on. Code: https://github.com/gulnazaki/counterfactual-benchmark.