# What is Healthy? Generative Counterfactual Diffusion for Lesion Localization

arXiv: https://arxiv.org/abs/2207.12268 · HTML: https://mlsanity.com/paper/2207.12268/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 19 of 19)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Diffusion Models for Medical Anomaly Detection](https://mlsanity.com/paper/2203.04306/) (2022)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study](https://mlsanity.com/paper/2004.03271/) (2020)
- [Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models](https://mlsanity.com/paper/2206.03461/) (2022)
- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [Unsupervised Anomaly Localization using Variational Auto-Encoders](https://mlsanity.com/paper/1907.02796/) (2019)
- [Unsupervised Lesion Detection via Image Restoration with a Normative Prior](https://mlsanity.com/paper/2005.00031/) (2020)
- [Context-encoding Variational Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1812.05941/) (2018)
- [Challenging Current Semi-Supervised Anomaly Segmentation Methods for Brain MRI](https://mlsanity.com/paper/2109.06023/) (2021)
- [Pseudo-healthy synthesis with pathology disentanglement and adversarial learning](https://mlsanity.com/paper/2005.01607/) (2020)
- [Unsupervised anomaly localization using VAE and beta-VAE](https://mlsanity.com/paper/2005.10686/) (2020)

## Built on by (most cited first, 30 of 52)

- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [Generative AI for Medical Imaging: extending the MONAI Framework](https://mlsanity.com/paper/2307.15208/) (2023)
- [High Fidelity Image Counterfactuals with Probabilistic Causal Models](https://mlsanity.com/paper/2306.15764/) (2023)
- [Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images](https://mlsanity.com/paper/2301.04802/) (2023)
- [Patched Diffusion Models for Unsupervised Anomaly Detection in Brain MRI](https://mlsanity.com/paper/2303.03758/) (2023)
- [RadEdit: stress-testing biomedical vision models via diffusion image editing](https://mlsanity.com/paper/2312.12865/) (2023)
- [Unsupervised Anomaly Detection in Medical Images Using Masked Diffusion Model](https://mlsanity.com/paper/2305.19867/) (2023)
- [The role of noise in denoising models for anomaly detection in medical images](https://mlsanity.com/paper/2301.08330/) (2023)
- [Benchmarking Counterfactual Image Generation](https://mlsanity.com/paper/2403.20287/) (2024)
- [Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models](https://mlsanity.com/paper/2306.01322/) (2023)
- [Guided Reconstruction with Conditioned Diffusion Models for Unsupervised Anomaly Detection in Brain MRIs](https://mlsanity.com/paper/2312.04215/) (2023)
- [DreaMR: Diffusion-driven Counterfactual Explanation for Functional MRI](https://mlsanity.com/paper/2307.09547/) (2023)
- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [Diffusion-based Iterative Counterfactual Explanations for Fetal Ultrasound Image Quality Assessment](https://mlsanity.com/paper/2403.08700/) (2024)
- [Counterfactual contrastive learning: robust representations via causal image synthesis](https://mlsanity.com/paper/2403.09605/) (2024)
- [CardiacNet: Learning to Reconstruct Abnormalities for Cardiac Disease Assessment from Echocardiogram Videos](https://mlsanity.com/paper/2410.20769/) (2024)
- [Emerging Synergies in Causality and Deep Generative Models: A Survey](https://mlsanity.com/paper/2301.12351/) (2023)
- [Counterfactual Explanations for Medical Image Classification and Regression using Diffusion Autoencoder](https://mlsanity.com/paper/2408.01571/) (2024)
- [Diffusion Counterfactual Generation with Semantic Abduction](https://mlsanity.com/paper/2506.07883/) (2025)
- [Beyond Known Reality: Exploiting Counterfactual Explanations for Medical Research](https://mlsanity.com/paper/2307.02131/) (2023)
- [A Survey of Emerging Applications of Diffusion Probabilistic Models in MRI](https://mlsanity.com/paper/2311.11383/) (2023)
- [Conditional Diffusion Models are Medical Image Classifiers that Provide Explainability and Uncertainty for Free](https://mlsanity.com/paper/2502.03687/) (2025)
- [Diff-Def: Diffusion-Generated Deformation Fields for Conditional Atlases](https://mlsanity.com/paper/2403.16776/) (2024)
- [Generating Counterfactual Trajectories with Latent Diffusion Models for Concept Discovery](https://mlsanity.com/paper/2404.10356/) (2024)
- [MedEdit: Counterfactual Diffusion-based Image Editing on Brain MRI](https://mlsanity.com/paper/2407.15270/) (2024)
- [Binary Noise for Binary Tasks: Masked Bernoulli Diffusion for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2403.11667/) (2024)
- [Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models](https://mlsanity.com/paper/2404.12920/) (2024)

## Abstract (arXiv)

Reducing the requirement for densely annotated masks in medical image segmentation is important due to cost constraints. In this paper, we consider the problem of inferring pixel-level predictions of brain lesions by only using image-level labels for training. By leveraging recent advances in generative diffusion probabilistic models (DPM), we synthesize counterfactuals of "How would a patient appear if X pathology was not present?". The difference image between the observed patient state and the healthy counterfactual can be used for inferring the location of pathology. We generate counterfactuals that correspond to the minimal change of the input such that it is transformed to healthy domain. This requires training with healthy and unhealthy data in DPMs. We improve on previous counterfactual DPMs by manipulating the generation process with implicit guidance along with attention conditioning instead of using classifiers. Code is available at https://github.com/vios-s/Diff-SCM.