# Counterfactual Image Synthesis for Discovery of Personalized Predictive Image Markers

arXiv: https://arxiv.org/abs/2208.02311 · HTML: https://mlsanity.com/paper/2208.02311/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 3 Aug 2022.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Visual Feature Attribution using Wasserstein GANs](https://mlsanity.com/paper/1711.08998/) (2017)
- [Learning Fixed Points in Generative Adversarial Networks: From Image-to-Image Translation to Disease Detection and Localization](https://mlsanity.com/paper/1908.06965/) (2019)
- [Conditional Generative Models for Counterfactual Explanations](https://mlsanity.com/paper/2101.10123/) (2021)
- [A Structural Causal Model for MR Images of Multiple Sclerosis](https://mlsanity.com/paper/2103.03158/) (2021)
- [End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification](https://mlsanity.com/paper/1810.00523/) (2018)
- [Adversarial Pseudo Healthy Synthesis Needs Pathology Factorization](https://mlsanity.com/paper/1901.07295/) (2019)
- [Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays](https://mlsanity.com/paper/2102.09475/) (2021)

## Built on by (most cited first, 10 of 10)

- [Debiasing Counterfactuals In the Presence of Spurious Correlations](https://mlsanity.com/paper/2308.10984/) (2023)
- [Counterfactual Explanations for Medical Image Classification and Regression using Diffusion Autoencoder](https://mlsanity.com/paper/2408.01571/) (2024)
- [RL4Med-DDPO: Reinforcement Learning for Controlled Guidance Towards Diverse Medical Image Generation using Vision-Language Foundation Models](https://mlsanity.com/paper/2503.15784/) (2025)
- [Explaining 3D Computed Tomography Classifiers with Counterfactuals](https://mlsanity.com/paper/2502.07156/) (2025)
- [Temporally Adjustable Longitudinal Fluid-Attenuated Inversion Recovery MRI Estimation / Synthesis for Multiple Sclerosis](https://mlsanity.com/paper/2209.04275/) (2022)
- [Causal Image Synthesis of Brain MR in 3D](https://mlsanity.com/paper/2303.14349/) (2023)
- [Longitudinal Causal Image Synthesis](https://mlsanity.com/paper/2410.17691/) (2024)
- [Pixel Perfect MegaMed: A Megapixel-Scale Vision-Language Foundation Model for Generating High Resolution Medical Images](https://mlsanity.com/paper/2507.12698/) (2025)
- [Interpretability and Individuality in Knee MRI: Patient-Specific Radiomic Fingerprint with Reconstructed Healthy Personas](https://mlsanity.com/paper/2601.08604/) (2026)
- [A continually expandable foundation model for brain MRI](https://mlsanity.com/paper/2608.08319/) (2026)

## Abstract (arXiv)

The discovery of patient-specific imaging markers that are predictive of future disease outcomes can help us better understand individual-level heterogeneity of disease evolution. In fact, deep learning models that can provide data-driven personalized markers are much more likely to be adopted in medical practice. In this work, we demonstrate that data-driven biomarker discovery can be achieved through a counterfactual synthesis process. We show how a deep conditional generative model can be used to perturb local imaging features in baseline images that are pertinent to subject-specific future disease evolution and result in a counterfactual image that is expected to have a different future outcome. Candidate biomarkers, therefore, result from examining the set of features that are perturbed in this process. Through several experiments on a large-scale, multi-scanner, multi-center multiple sclerosis (MS) clinical trial magnetic resonance imaging (MRI) dataset of relapsing-remitting (RRMS) patients, we demonstrate that our model produces counterfactuals with changes in imaging features that reflect established clinical markers predictive of future MRI lesional activity at the population level. Additional qualitative results illustrate that our model has the potential to discover novel and subject-specific predictive markers of future activity.