# Towards Interpretable Counterfactual Generation via Multimodal Autoregression

arXiv: https://arxiv.org/abs/2503.23149 · HTML: https://mlsanity.com/paper/2503.23149/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2025.

## Built on by (most cited first, 11 of 11)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis](https://mlsanity.com/paper/2510.15710/) (2025)
- [Mask-Guided Attention Regulation for Anatomically Consistent Counterfactual CXR Synthesis](https://mlsanity.com/paper/2603.04130/) (2026)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [Integrating Anatomical Priors into a Causal Diffusion Model](https://mlsanity.com/paper/2509.09054/) (2025)
- [Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering](https://mlsanity.com/paper/2602.23676/) (2026)
- [CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging](https://mlsanity.com/paper/2604.22989/) (2026)
- [Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI](https://mlsanity.com/paper/2604.05171/) (2026)
- [Learning Directional Semantic Transitions for Longitudinal Chest X-ray Analysis](https://mlsanity.com/paper/2606.15938/) (2026)
- [State Space Models Meet Remote Sensing: A Survey](https://mlsanity.com/paper/2606.25329/) (2026)
- [MedUAG: Unified Understanding and Generation for Medical Multimodal Models](https://mlsanity.com/paper/2608.18937/) (2026)

## Abstract (arXiv)

Counterfactual medical image generation enables clinicians to explore clinical hypotheses, such as predicting disease progression, facilitating their decision-making. While existing methods can generate visually plausible images from disease progression prompts, they produce silent predictions that lack interpretation to verify how the generation reflects the hypothesized progression -- a critical gap for medical applications that require traceable reasoning. In this paper, we propose Interpretable Counterfactual Generation (ICG), a novel task requiring the joint generation of counterfactual images that reflect the clinical hypothesis and interpretation texts that outline the visual changes induced by the hypothesis. To enable ICG, we present ICG-CXR, the first dataset pairing longitudinal medical images with hypothetical progression prompts and textual interpretations. We further introduce ProgEmu, an autoregressive model that unifies the generation of counterfactual images and textual interpretations. We demonstrate the superiority of ProgEmu in generating progression-aligned counterfactuals and interpretations, showing significant potential in enhancing clinical decision support and medical education. Project page: https://progemu.github.io.