mlsanityData to 7 Oct 2026

PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion

arXiv 2503.00196 ↗First posted 28 Feb 2025cs.CV cs.AIcited by 17

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Developing reliable and generalizable deep learning systems for medical imaging faces significant obstacles due to spurious correlations, data imbalances, and limited text annotations in datasets. Addressing these challenges requires architectures that are robust to the unique complexities posed by medical imaging data. Rapid advancements in vision-language foundation models within the natural image domain prompt the question of how they can be adapted for medical imaging tasks. In this work, we present PRISM, a framework that leverages foundation models to generate high-resolution, language-guided medical image counterfactuals using Stable Diffusion. Our approach demonstrates unprecedented precision in selectively modifying spurious correlations (the medical devices) and disease features, enabling the removal and addition of specific attributes while preserving other image characteristics. Through extensive evaluation, we show how PRISM advances counterfactual generation and enables the development of more robust downstream classifiers for clinically deployable solutions. To facilitate broader adoption and research, we make our code publicly available at https://github.com/Amarkr1/PRISM.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 27 of its 34 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

27 of 34 references.

References · 27

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  7. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  8. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 612
  11. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  12. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 224
15 more
  1. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  2. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  3. 2019Explanation by Progressive Exaggerationcited by 76
  4. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  5. 2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 54
  6. 2023High Fidelity Image Counterfactuals with Probabilistic Causal Modelscited by 53
  7. 2020GANterfactual - Counterfactual Explanations for Medical Non-Experts using Generative Adversarial Learningcited by 50
  8. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  9. 2023MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classificationcited by 38
  10. 2020CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Netscited by 29
  11. 2023BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeyscited by 29
  12. 2023Debiasing Counterfactuals In the Presence of Spurious Correlationscited by 12
  13. 2024Low-resource finetuning of foundation models beats state-of-the-art in histopathologycited by 10
  14. 2024DeViDe: Faceted medical knowledge for improved medical vision-language pre-trainingcited by 10
  15. 2024DeCoDEx: Confounder Detector Guidance for Improved Diffusion-based Counterfactual Explanationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 8.

17 papers.

Sort:

By area:

  1. 2025RL4Med-DDPO: Reinforcement Learning for Controlled Guidance Towards Diverse Medical Image Generation using Vision-Language Foundation Modelscited by 5
  2. 2025Counterfactual Identifiability via Dynamic Optimal Transportcited by 5
  3. 2025Discovering Latent Graphs with GFlowNets for Diverse Conditional Image Generationcited by 2
  4. 2026Scaling Generative Foundation Models for Chest Radiography with Rectified Flow Transformerscited by 2
  5. 2025Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imagingcited by 1
  6. 2025Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Modelscited by 1
  7. 2025RoentMod: A Synthetic Chest X-Ray Modification Model to Identify and Correct Image Interpretation Model Shortcutscited by 1
  8. 2025Med-Banana: Learning Quality-Controlled Medical Image Editing from Success-and-Failure Trajectoriescited by 1
  9. 2026When the Edit Changes the Patient: Measuring Identity Preservation in Counterfactual Retinal Imagescited by 1
  10. 2026Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Modelscited by 1
  11. 2025Language-Guided Trajectory Traversal in Disentangled Stable Diffusion Latent Space for Factorized Medical Image Generationcited by 0
  12. 2025Pixel Perfect MegaMed: A Megapixel-Scale Vision-Language Foundation Model for Generating High Resolution Medical Imagescited by 0
5 more
  1. 2026MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Promptscited by 0
  2. 2026RED-Sphere: Hyperspherical Residual Edge Debiasing for Cross-Population Fundus Disease Domain Generalizationcited by 0
  3. 2026Symbal: Detecting Systematic Misalignments in Model-Generated Captionscited by 0
  4. 2026FiRe: Fixed-Noise Refinement for Visual Counterfactual Explanationscited by 0
  5. 2026Spatially Gated Diffusion for Localized Counterfactual Chest Radiograph Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Imagining Alternatives: Towards High-Resolution 3D Counterfactual Medical Image Generation via Language Guidance
  2. 2026MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation
  3. 2023Generating Realistic Counterfactuals for Retinal Fundus and OCT Images using Diffusion Models
  4. 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains
  5. 2026Generating Medical Image Counterfactuals using Causal Explanations
  6. 2026Compositional Reward Models for Conditional Medical Image Generation
  7. 2025Causal Disentanglement for Robust Long-tail Medical Image Generation
  8. 2022What is Healthy? Generative Counterfactual Diffusion for Lesion Localization