UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis
Not on the map. Why?
Abstract (arXiv)
Medical workflows routinely combine reading images with producing visual and textual outputs, making both image understanding and generation central to medical AI. Most existing systems, however, address these abilities in isolated models, losing the shared knowledge that a unified architecture could exploit. To bridge this gap, we present UniMedVL, the first unified medical model that seamlessly integrates multimodal understanding and generation capabilities within a single model without switching weights. We achieve this via a tailored progressive training pipeline where understanding and generation mutually reinforce each other. To effectively train UniMedVL, we curate UniMedVL-5M, the first large-scale medical dataset comprising over 5.6M instances across 8 medical imaging modalities, tailored for multimodal input-output tasks in unified medical understanding and generation. Experimental results demonstrate that UniMedVL achieves competitive performance on five medical understanding benchmarks. Crucially, UniMedVL natively supports diverse interleaved generation tasks, e.g., virtual staining, super-resolution, cross-modal synthesis, essential for complex medical workflows. Our code and dataset are publicly available.
Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 86 of its 116 references that are in the map. First posted on arXiv 17 Oct 2025.
Builds on
86 of 116 references.
References · 86
- 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
- 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
- 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
- 2014Image Super-Resolution Using Deep Convolutional Networkscited by 1,218
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
- 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2015Accurate Image Super-Resolution Using Very Deep Convolutional Networkscited by 902
38 more
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
- 2017Unsupervised Image-to-Image Translation Networkscited by 602
- 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
- 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
- 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
- 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
- 2025MedGemma Technical Reportcited by 398
- 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
- 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
- 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
- 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
- 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
- 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
- 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
- 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
- 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
- 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
- 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
- 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
- 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
- 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
- 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
- 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 100
- 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
- 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
- 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
- 2022Integrated multimodal artificial intelligence framework for healthcare applicationscited by 69
- 2022BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pixcited by 61
- 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Modelscited by 61
- 2024SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understandingcited by 54
50 most cited of 86 All 86 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 10.
12 papers.
- 2026MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learningcited by 6
- 2025Beyond Pixel Simulation: Pathology Image Generation via Diagnostic Semantic Tokens and Prototype Controlcited by 3
- 2026Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Spacecited by 3
- 2026Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMscited by 3
- 2025Omni-Weather: A Unified Multimodal Model for Weather Radar Understanding and Generationcited by 2
- 2026MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmarkcited by 1
- 2026MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradationscited by 0
- 2026MedQ-UNI: Toward Unified Medical Image Quality Assessment and Restoration via Vision-Language Modelingcited by 0
- 2026SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignmentcited by 0
- 2026Unified Multimodal Model for Brain MRI Imputation and Understandingcited by 0
- 2026Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomographycited by 0
- 2026MedUAG: Unified Understanding and Generation for Medical Multimodal Modelscited by 0
- 2026MedUAG: Unified Understanding and Generation for Medical Multimodal Models19 Aug 2026
- 2026Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography28 Jul 2026
- 2026Unified Multimodal Model for Brain MRI Imputation and Understanding15 Jun 2026
- 2026SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment9 May 2026
- 2026MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark12 Apr 2026
- 2026Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs21 Mar 2026
- 2026MedQ-UNI: Toward Unified Medical Image Quality Assessment and Restoration via Vision-Language Modeling19 Mar 2026
- 2026Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space14 Mar 2026
- 2026MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations8 Mar 2026
- 2026MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning3 Feb 2026
- 2025Omni-Weather: A Unified Multimodal Model for Weather Radar Understanding and Generation25 Dec 2025
- 2025Beyond Pixel Simulation: Pathology Image Generation via Diagnostic Semantic Tokens and Prototype Control24 Dec 2025