mlsanityData to 7 Oct 2026

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation

arXiv 2502.09838 ↗First posted 14 Feb 2025cs.CV cs.AIcited by 112

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

We present HealthGPT, a powerful Medical Large Vision-Language Model (Med-LVLM) that integrates medical visual comprehension and generation capabilities within a unified autoregressive paradigm. Our bootstrapping philosophy is to progressively adapt heterogeneous comprehension and generation knowledge to pre-trained large language models (LLMs). This is achieved through a novel heterogeneous low-rank adaptation (H-LoRA) technique, which is complemented by a tailored hierarchical visual perception approach and a three-stage learning strategy. To effectively learn the HealthGPT, we devise a comprehensive medical domain-specific comprehension and generation dataset called VL-Health. Experimental results demonstrate exceptional performance and scalability of HealthGPT in medical visual unified tasks. Our project can be accessed at https://github.com/DCDmllm/HealthGPT.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 47 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

35 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge19 Nov
  2. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
  3. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  11. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
19 more
  1. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  2. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  3. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  4. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  5. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  6. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  7. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  8. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  9. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  11. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  12. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  13. 2022BBDM: Image-to-image Translation with Brownian Bridge Diffusion Modelscited by 222
  14. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  15. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  16. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  17. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  18. 2023SynthRAD2023 Grand Challenge dataset: generating synthetic CT for radiotherapycited by 53
  19. 2024Soft Masked Mamba Diffusion Model for CT to MRI Conversioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 52; 2026: 60.

112 papers.

Sort:

By area:

  1. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  3. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  4. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  5. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  6. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 20
  7. 2025Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysiscited by 16
  8. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  9. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  10. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 12
  11. 2025DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?cited by 10
  12. 2025MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precisioncited by 10
38 more
  1. 2025OralGPT-Omni: A Versatile Dental Multimodal Large Language Modelcited by 10
  2. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbationscited by 9
  3. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  4. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 9
  5. 2025MedGR2: Breaking the Data Barrier for Medical Reasoning via Generative Reward Learningcited by 8
  6. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  7. 2025Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoningcited by 8
  8. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  9. 2025Tree-of-Reasoning: Towards Complex Medical Diagnosis via Multi-Agent Reasoning with Evidence Treecited by 7
  10. 2025MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Modelscited by 6
  11. 2025Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoningcited by 6
  12. 2026M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understandingcited by 6
  13. 2026CtrlCoT: Dual-Granularity Chain-of-Thought Compression for Controllable Reasoningcited by 6
  14. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  15. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  16. 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
  17. 2025CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographscited by 5
  18. 2025CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rayscited by 5
  19. 2025DentalGPT: Incentivizing Multimodal Complex Reasoning in Dentistrycited by 5
  20. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compressioncited by 5
  21. 2025RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivitycited by 4
  22. 2025Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Modelscited by 4
  23. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentationcited by 4
  24. 2026OralGPT-Plus: Learning to Use Visual Tools via Reinforcement Learning for Panoramic X-ray Analysiscited by 4
  25. 2026VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policiescited by 4
  26. 2025HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understandingcited by 3
  27. 2025Generative AI for Healthcare: Fundamentals, Challenges, and Perspectivescited by 3
  28. 2026Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Benchcited by 3
  29. 2026MMedExpert-R1: Strengthening Multimodal Medical Reasoning via Domain-Specific Adaptation and Clinical Guideline Reinforcementcited by 3
  30. 2026UniX: Unifying Autoregression and Diffusion for Chest X-Ray Understanding and Generationcited by 3
  31. 2026CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Frameworkcited by 3
  32. 2026Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Spacecited by 3
  33. 2026Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMscited by 3
  34. 2026Joint Optimization of Reasoning and Dual-Memory for Self-Learning Diagnostic Agentcited by 3
  35. 2026OralAgent: Integrating Reasoning, Tools, and Knowledge for Interactive Dental Image Analysiscited by 3
  36. 2026Can Multimodal Large Language Models Understand OCT?cited by 3
  37. 2025MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Bookcited by 2
  38. 2025EH-Benchmark Ophthalmic Hallucination Benchmark and Agent-Driven Top-Down Traceable Reasoning Workflowcited by 2

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
  2. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  3. 2025Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding
  4. 2024Advancing High Resolution Vision-Language Models in Biomedicine
  5. 2024Beyond the Hype: A dispassionate look at vision-language models in medical scenario
  6. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  7. 2025Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis
  8. 2024Can LLMs' Tuning Methods Work in Medical Multimodal Domain?