mlsanityData to 7 Oct 2026

Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains

arXiv 2210.04133 ↗First posted 9 Oct 2022cs.CV cs.AI cs.CLcited by 85

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 1 more
Abstract (arXiv)

Multi-modal foundation models are typically trained on millions of pairs of natural images and text captions, frequently obtained through web-crawling approaches. Although such models depict excellent generative capabilities, they do not typically generalize well to specific domains such as medical images that have fundamentally shifted distributions compared to natural images. Building generative models for medical images that faithfully depict clinical context may help alleviate the paucity of healthcare datasets. Thus, in this study, we seek to research and expand the representational capabilities of large pretrained foundation models to medical concepts, specifically for leveraging the Stable Diffusion model to generate domain specific images found in medical imaging. We explore the sub-components of the Stable Diffusion pipeline (the variational autoencoder, the U-Net and the text-encoder) to fine-tune the model to generate medical images. We benchmark the efficacy of these efforts using quantitative image quality metrics and qualitative radiologist-driven evaluations that accurately represent the clinical content of conditional text prompts. Our best-performing model improves upon the stable diffusion baseline and can be conditioned to insert a realistic-looking abnormality on a synthetic radiology image, while maintaining a 95% accuracy on a classifier trained to detect the abnormality.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 22 references that are in the map. First posted on arXiv 9 Oct 2022.

Builds on

18 of 22 references.

References · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  12. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
6 more
  1. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  2. 2022Brain Imaging Generation with Latent Diffusion Modelscited by 267
  3. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  4. 2017GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibriumcited by 208
  5. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  6. 2020On the limits of cross-domain generalization in automated X-ray predictioncited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 37; 2024: 29; 2025: 13; 2026: 3.

85 papers.

Sort:

By area:

  1. 2023Extracting Training Data from Diffusion Modelscited by 723
  2. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  3. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  4. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  5. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  6. 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generationcited by 89
  7. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 85
  8. 2023CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Imagescited by 84
  9. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  10. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  11. 2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
  12. 2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 54
38 more
  1. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  2. 2023Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Imagescited by 45
  3. 2022Generation of Anonymous Chest Radiographs Using Latent Diffusion Models for Training Thoracic Abnormality Classification Systemscited by 44
  4. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  5. 2023A Pathway Towards Responsible AI Generated Contentcited by 38
  6. 2023Cascaded Latent Diffusion Models for High-Resolution Chest X-ray Synthesiscited by 32
  7. 2024Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomographycited by 31
  8. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  9. 2023Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarkscited by 30
  10. 2023BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeyscited by 29
  11. 2023DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathologycited by 28
  12. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  13. 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 25
  14. 2023Augmenting medical image classifiers with synthetic data from latent diffusion modelscited by 21
  15. 2023Medical diffusion on a budget: Textual Inversion for medical image generationcited by 20
  16. 2025MedVAE: Efficient Automated Interpretation of Medical Images with Large-Scale Generalizable Autoencoderscited by 20
  17. 2024SurGen: Text-Guided Diffusion Model for Surgical Video Generationcited by 19
  18. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusioncited by 17
  19. 2024Chest-Diffusion: A Light-Weight Text-to-Image Model for Report-to-CXR Generationcited by 16
  20. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  21. 2023FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answeringcited by 15
  22. 2023Vision-Language Generative Model for View-Specific Chest X-ray Generationcited by 14
  23. 2023Aligning Synthetic Medical Images with Clinical Knowledge using Human Feedbackcited by 14
  24. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  25. 2023AI-Generated Images as Data Source: The Dawn of Synthetic Eracited by 13
  26. 2024Uncovering the Text Embedding in Text-to-Image Diffusion Modelscited by 13
  27. 2024Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-trainingcited by 13
  28. 2024Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesiscited by 13
  29. 2023Towards Equitable Representation in Text-to-Image Synthesis Models with the Cross-Cultural Understanding Benchmark (CCUB) Datasetcited by 12
  30. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  31. 2023MAM-E: Mammographic synthetic image generation with diffusion modelscited by 11
  32. 2024Towards Learning Contrast Kinetics with Multi-Condition Latent Diffusion Modelscited by 11
  33. 2024Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Promptingcited by 10
  34. 2023Word-Level Explanations for Analyzing Bias in Text-to-Image Modelscited by 8
  35. 2023Diverse Diffusion: Enhancing Image Diversity in Text-to-Image Generationcited by 8
  36. 2024XReal: Realistic Anatomy and Pathology-Aware X-ray Generation via Controllable Diffusion Modelcited by 8
  37. 2023Navigating the Synthetic Realm: Harnessing Diffusion-based Models for Laparoscopic Text-to-Image Generationcited by 7
  38. 2024FairQueue: Rethinking Prompt Learning for Fair Text-to-Image Generationcited by 7

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Models
  2. 2025Language-Guided Trajectory Traversal in Disentangled Stable Diffusion Latent Space for Factorized Medical Image Generation
  3. 2025Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation
  4. 2025Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation
  5. 2024A Generalist Model for Diverse Text-Guided Medical Image Synthesis
  6. 2023Trade-offs in Fine-tuned Diffusion Models Between Accuracy and Interpretability
  7. 2023Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images
  8. 2024MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant