mlsanityData to 7 Oct 2026

GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI

arXiv 2411.14522 ↗First posted 21 Nov 2024cs.CVcited by 23

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Despite significant advancements in general AI, its effectiveness in the medical domain is limited by the lack of specialized medical knowledge. To address this, we formulate GMAI-VL-5.5M, a multimodal medical dataset created by converting hundreds of specialized medical datasets with various annotations into high-quality image-text pairs. This dataset offers comprehensive task coverage, diverse modalities, and rich image-text data. Building upon this dataset, we develop GMAI-VL, a general medical vision-language model, with a three-stage training strategy that enhances the integration of visual and textual information. This approach significantly improves the model's ability to process multimodal data, supporting accurate diagnoses and clinical decision-making. Experiments show that GMAI-VL achieves state-of-the-art performance across various multimodal medical tasks, including visual question answering and medical image diagnosis.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 73 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

49 of 73 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI6 Aug
  2. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine6 Aug
  3. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
  4. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  5. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare27 Oct
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  7. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
30 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2016A Diagram Is Worth A Dozen Imagescited by 793
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  9. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  12. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  13. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  14. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  15. 2021OCR-free Document Understanding Transformercited by 400
  16. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  17. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  18. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  19. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  20. 2023Towards Generalist Biomedical AIcited by 286
  21. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  22. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  23. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  24. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  25. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  26. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  27. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  28. 2023SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million maskscited by 54
  29. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  30. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 7.

23 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  3. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  4. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  5. 2025MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosiscited by 32
  6. 2025Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Searchcited by 16
  7. 2025Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysiscited by 16
  8. 2025MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbookscited by 14
  9. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  10. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 12
  11. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  12. 2025OralGPT-Omni: A Versatile Dental Multimodal Large Language Modelcited by 10
11 more
  1. 2025SciReasoner: Laying the Scientific Reasoning Ground Across Disciplinescited by 9
  2. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  3. 2025Constructing Ophthalmic MLLM for Positioning-diagnosis Collaboration Through Clinical Cognitive Chain Reasoningcited by 6
  4. 2026Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Benchcited by 3
  5. 2026MMedExpert-R1: Strengthening Multimodal Medical Reasoning via Domain-Specific Adaptation and Clinical Guideline Reinforcementcited by 3
  6. 2025Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMscited by 2
  7. 2025Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Datacited by 1
  8. 2025Zero-Shot Multi-modal Large Language Model v.s. Supervised Deep Learning: A Comparative Study on CT-Based Intracranial Hemorrhage Subtypingcited by 1
  9. 2026Aloe-Vision: Robust Vision-Language Models for Healthcarecited by 1
  10. 2026Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysiscited by 0
  11. 2026InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation
  2. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs
  3. 2025Vision Language Models in Medicine
  4. 2025Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI
  5. 2024Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
  6. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
  7. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  8. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound