mlsanityData to 14 Oct 2026

Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI

arXiv 2508.08270 ↗First posted 30 Jul 2025cs.LG cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large multimodal models (LMMs) have demonstrated significant potential in providing innovative solutions for various biomedical tasks, including pathology analysis, radiology report generation, and biomedical assistance. However, the existing multimodal biomedical AI is typically based on foundation LLMs, thus hindering the understanding of intricate medical concepts with limited medical training data. Moreover, recent LLaVA-induced medical LMMs struggle to effectively capture the intricate relationship between the texts and the images. Therefore, we introduce Doctor Sun, a large multimodal generative model specialized in medicine, developed to encode, integrate, and interpret diverse biomedical data modalities such as text and images. In particular, Doctor Sun integrates a pre-trained vision encoder with a medical LLM and conducts two-stage training on various medical datasets, focusing on feature alignment and instruction tuning. Moreover, we release SunMed-VL, a wide-range bilingual medical multimodal dataset, along with all associated models, code, and resources, to freely support the advancement of biomedical multimodal research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 59 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

34 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models14 Feb
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  3. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May

Other references · 31

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  7. 2024Mixtral of Expertscited by 1,823
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
19 more
  1. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  2. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  3. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  4. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  5. 2024What matters when building vision-language models?cited by 326
  6. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  7. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  8. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 214
  9. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  10. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  11. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 162
  12. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  13. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 131
  14. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
  15. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 115
  16. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  17. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 90
  18. 2018The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesionscited by 26
  19. 2023A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Modelcited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
  2. 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation
  3. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study
  4. 2025Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis
  5. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  6. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks
  7. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models
  8. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images