mlsanityData to 14 Oct 2026

Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insights

arXiv 2506.17337 ↗First posted 19 Jun 2025eess.IV cs.AI cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision Language Models (VLMs) have shown promise in automating image diagnosis and interpretation in clinical settings. However, developing specialist medical VLMs requires substantial computational resources and carefully curated datasets, and it remains unclear under which conditions generalist and specialist medical VLMs each perform best. This study highlights the complementary strengths of specialist medical and generalist VLMs. Specialists remain valuable in modality-aligned use cases, but we find that efficiently fine-tuned generalist VLMs can achieve comparable or even superior performance in most tasks, particularly when transferring to unseen or rare OOD medical modalities. These results suggest that generalist VLMs, rather than being constrained by their lack of specialist medical pretraining, may offer a scalable and cost-effective pathway for advancing clinical AI development.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 73 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

39 of 73 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning8 Jun
  2. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge19 Nov
  3. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?6 Nov
  4. 2024LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models2 Oct
  5. 2024Beyond the Hype: A dispassionate look at vision-language models in medical scenario16 Aug
  6. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models14 Feb
  7. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  9. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  12. 2025Gemma 3 Technical Reportcited by 1,824
18 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  2. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  5. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  6. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  7. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  8. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  9. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  10. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
  11. 2019Causality matters in medical imagingcited by 193
  12. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  13. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  14. 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
  15. 2023Harvard Glaucoma Fairness: A Retinal Nerve Disease Dataset for Fairness Learning and Fair Identity Normalizationcited by 31
  16. 2018The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesionscited by 26
  17. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  18. 2024MedMobile: A mobile-sized language model with clinical capabilitiescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Verification Mirage: Mapping the Reliability Boundary of Self-Verification in Medical VQAcited by 2
  2. 2026EG-ARSA: An Expert-Grounded Open Model for Visual Road Safety Auditing in Low-Resource Settingscited by 0
  3. 2026MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answeringcited by 0
  4. 2026Multimodal Medical Image Parsing with Task-Specific Low-Rank Adapterscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  2. 2025Vision Language Models in Medicine
  3. 2026How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
  4. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  5. 2024Specialized curricula for training vision-language models in retinal image analysis
  6. 2026Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysis
  7. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  8. 2023Towards General Purpose Medical AI: Continual Learning Medical Foundation Model