# Diffusion Models for Counterfactual Generation and Anomaly Detection in Brain Images

arXiv: https://arxiv.org/abs/2308.02062 · HTML: https://mlsanity.com/paper/2308.02062/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 33 of its 58 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 33)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Diffusion Models for Medical Anomaly Detection](https://mlsanity.com/paper/2203.04306/) (2022)
- [Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge](https://mlsanity.com/paper/1904.00682/) (2019)
- [Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders](https://mlsanity.com/paper/1806.04972/) (2018)
- [How good is my GAN?](https://mlsanity.com/paper/1807.09499/) (2018)
- [Context-encoding Variational Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1812.05941/) (2018)
- [Mask, Stitch, and Re-Sample: Enhancing Robustness and Generalizability in Anomaly Detection through Automatic Diffusion Models](https://mlsanity.com/paper/2305.19643/) (2023)
- [Visual Feature Attribution using Wasserstein GANs](https://mlsanity.com/paper/1711.08998/) (2017)
- [CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysis](https://mlsanity.com/paper/2001.05149/) (2020)
- [Learning Fixed Points in Generative Adversarial Networks: From Image-to-Image Translation to Disease Detection and Localization](https://mlsanity.com/paper/1908.06965/) (2019)
- [Using StyleGAN for Visual Interpretability of Deep Learning Models on Medical Images](https://mlsanity.com/paper/2101.07563/) (2021)
- [Testing the robustness of attribution methods for convolutional neural networks in MRI-based Alzheimer's disease classification](https://mlsanity.com/paper/1909.08856/) (2019)

## Built on by (most cited first, 30 of 30)

- [RadEdit: stress-testing biomedical vision models via diffusion image editing](https://mlsanity.com/paper/2312.12865/) (2023)
- [Benchmarking Counterfactual Image Generation](https://mlsanity.com/paper/2403.20287/) (2024)
- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Generating Compositional Scenes via Text-to-image RGBA Instance Generation](https://mlsanity.com/paper/2411.10913/) (2024)
- [A Survey on Diffusion Models for Anomaly Detection](https://mlsanity.com/paper/2501.11430/) (2025)
- [Counterfactual contrastive learning: robust representations via causal image synthesis](https://mlsanity.com/paper/2403.09605/) (2024)
- [Counterfactual Explanations for Medical Image Classification and Regression using Diffusion Autoencoder](https://mlsanity.com/paper/2408.01571/) (2024)
- [Diffusion-empowered AutoPrompt MedSAM](https://mlsanity.com/paper/2502.06817/) (2025)
- [Anomaly Detection and Generation with Diffusion Models: A Survey](https://mlsanity.com/paper/2506.09368/) (2025)
- [Probing the Limits and Capabilities of Diffusion Models for the Anatomic Editing of Digital Twins](https://mlsanity.com/paper/2401.00247/) (2023)
- [MAEDiff: Masked Autoencoder-enhanced Diffusion Models for Unsupervised Anomaly Detection in Brain Images](https://mlsanity.com/paper/2401.10561/) (2024)
- [Applications of Large Models in Medicine](https://mlsanity.com/paper/2502.17132/) (2025)
- [Causally Steered Diffusion for Automated Video Counterfactual Generation](https://mlsanity.com/paper/2506.14404/) (2025)
- [Direct Distributional Optimization for Provable Alignment of Diffusion Models](https://mlsanity.com/paper/2502.02954/) (2025)
- [Counterfactual Probabilistic Diffusion with Expert Models](https://mlsanity.com/paper/2508.13355/) (2025)
- [Denoising Diffusion Models for Anomaly Localization in Medical Images](https://mlsanity.com/paper/2410.23834/) (2024)
- [Reimagining Anomalies: What If Anomalies Were Normal?](https://mlsanity.com/paper/2402.14469/) (2024)
- [DiCoFlex: Model-agnostic diverse counterfactuals with flexible control](https://mlsanity.com/paper/2505.23700/) (2025)
- [WDT-MD: Wavelet Diffusion Transformers for Microaneurysm Detection in Fundus Images](https://mlsanity.com/paper/2511.08987/) (2025)
- [Early Lung Cancer Diagnosis from Virtual Follow-up LDCT Generation via Correlational Autoencoder and Latent Flow Matching](https://mlsanity.com/paper/2511.18185/) (2025)
- [Statistical Test for Diffusion-Based Anomaly Localization via Selective Inference](https://mlsanity.com/paper/2402.11789/) (2024)
- [Point-Based Shape Representation Generation with a Correspondence-Preserving Diffusion Model](https://mlsanity.com/paper/2508.03925/) (2025)
- [Unsupervised Deep Generative Models for Anomaly Detection in Neuroimaging: A Systematic Scoping Review](https://mlsanity.com/paper/2510.14462/) (2025)
- [DCFO: Density-Based Counterfactuals for Outliers -- Additional Material](https://mlsanity.com/paper/2512.10659/) (2025)
- [An Interpretable Local Editing Model for Counterfactual Medical Image Generation](https://mlsanity.com/paper/2603.00423/) (2026)
- [When the Edit Changes the Patient: Measuring Identity Preservation in Counterfactual Retinal Images](https://mlsanity.com/paper/2608.23024/) (2026)
- [Development of a Deep Learning Method to Identify Acute Ischemic Stroke Lesions on Brain CT](https://mlsanity.com/paper/2309.17320/) (2023)
- [IgCONDA-PET: Weakly-Supervised PET Anomaly Detection using Implicitly-Guided Attention-Conditional Counterfactual Diffusion Modeling -- a Multi-Center, Multi-Cancer, and Multi-Tracer Study](https://mlsanity.com/paper/2405.00239/) (2024)
- [PolypSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering](https://mlsanity.com/paper/2603.07066/) (2026)
- [Reconstruction-Shift Discrimination via Mask-Guided Latent Diffusion for Medical Anomaly Detection](https://mlsanity.com/paper/2608.00444/) (2026)

## Abstract (arXiv)

Segmentation masks of pathological areas are useful in many medical applications, such as brain tumour and stroke management. Moreover, healthy counterfactuals of diseased images can be used to enhance radiologists' training files and to improve the interpretability of segmentation models. In this work, we present a weakly supervised method to generate a healthy version of a diseased image and then use it to obtain a pixel-wise anomaly map. To do so, we start by considering a saliency map that approximately covers the pathological areas, obtained with ACAT. Then, we propose a technique that allows to perform targeted modifications to these regions, while preserving the rest of the image. In particular, we employ a diffusion model trained on healthy samples and combine Denoising Diffusion Probabilistic Model (DDPM) and Denoising Diffusion Implicit Model (DDIM) at each step of the sampling process. DDPM is used to modify the areas affected by a lesion within the saliency map, while DDIM guarantees reconstruction of the normal anatomy outside of it. The two parts are also fused at each timestep, to guarantee the generation of a sample with a coherent appearance and a seamless transition between edited and unedited parts. We verify that when our method is applied to healthy samples, the input images are reconstructed without significant modifications. We compare our approach with alternative weakly supervised methods on the task of brain lesion segmentation, achieving the highest mean Dice and IoU scores among the models considered.