Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation
On the map: Diffusion models → Medical imaging and segmentation
Abstract (arXiv)
Recent advances in generative medical models are constrained by modality-specific scenarios that hinder the integration of complementary evidence from imaging, pathology, and clinical notes. This fragmentation limits their evolution into foundation models that can learn and reason across the full spectrum of biomedical data. We propose MeDiM, the first medical discrete diffusion model that learns shared distributions across modalities without modality-specific components. MeDiM unifies multiple generative tasks: translating between images and text, and jointly producing image-report pairs across domains in response to prompts. Built on a discrete diffusion framework, MeDiM bridges vision and language representations through a shared probabilistic space. To enable unified and flexible medical generation, we employ a multimodal large language model (MLLM) as the diffusion backbone, leveraging its prior knowledge and cross-modal reasoning. Two key designs are introduced: (1) removing the causal attention mask for bidirectional context, and (2) injecting continuous timestep embeddings for diffusion awareness. Experiments demonstrate high-fidelity medical generation (FID 16.60 on MIMIC-CXR and FID 24.19 on PathGen) and accurate report generation (METEOR 0.2650 and 0.2580). Jointly generated image-report pairs further enhance downstream performance (plus6.43 percent BLEU-1, plus18.57 percent BLEU-2, plus31.58 percent BLEU-3, plus4.80 percent METEOR), showing that MeDiM supports coherent and clinically grounded multimodal outputs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 55 of its 65 references that are in the map. First posted on arXiv 7 Oct 2025.
Builds on
55 of 65 references.
Closest ancestors · 6
Same direction, newest first
- 2025Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation25 Jun
- 2024Leapfrog Latent Consistency Model (LLCM) for Medical Images Generation22 Nov
- 2024Fast-DDPM: Fast Denoising Diffusion Probabilistic Models for Medical Image-to-Image Generation23 May
- 2024MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant7 Mar
- 2023DiffBoost: Enhancing Medical Image Segmentation via Text-Guided Diffusion Model19 Oct
- 2022Adaptive Diffusion Priors for Accelerated MRI Reconstruction12 Jul
Other references · 49
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2017Neural Discrete Representation Learningcited by 4,650
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
37 more
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
- 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
- 2022MaskGIT: Masked Generative Image Transformercited by 1,082
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2025Large Language Diffusion Modelscited by 872
- 2024Simple and Effective Masked Diffusion Language Modelscited by 800
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
- 2024KAN: Kolmogorov-Arnold Networkscited by 634
- 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
- 2023Generative Multimodal Models are In-Context Learnerscited by 446
- 2020Generating Radiology Reports via Memory-driven Transformercited by 363
- 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
- 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
- 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
- 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 176
- 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
- 2020When Radiology Report Generation Meets Knowledge Graphcited by 144
- 2023Planting a SEED of Vision in Large Language Modelcited by 140
- 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
- 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generationcited by 89
- 2022Conversion Between CT and MRI Images Using Diffusion and Score-Matching Modelscited by 69
- 2018SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truthcited by 68
- 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
- 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
- 2025Unified Multimodal Discrete Diffusioncited by 46
- 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generationcited by 39
- 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 39
- 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
- 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 25
- 2025MedSegFactory: Text-Guided Generation of Medical Image-Mask Pairscited by 13
- 2024PairAug: What Can Augmented Image-Text Pairs Do for Radiology?cited by 10
- 2024Factual Serialization Enhancement: A Key Innovation for Chest X-ray Report Generationcited by 10
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains
- 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
- 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images
- 2024Dual Diffusion for Unified Image Generation and Understanding
- 2025Language-Guided Trajectory Traversal in Disentangled Stable Diffusion Latent Space for Factorized Medical Image Generation
- 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
- 2025VAP-Diffusion: Enriching Descriptions with MLLMs for Enhanced Medical Image Generation
- 2024Mediffusion: Joint Diffusion for Self-Explainable Semi-Supervised Classification and Medical Image Generation