mlsanityData to 7 Oct 2026

On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study

arXiv 2402.14162 ↗First posted 21 Feb 2024cs.CV cs.AIcited by 28

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recently, large language models (LLMs) have taken the spotlight in natural language processing. Further, integrating LLMs with vision enables the users to explore emergent abilities with multimodal data. Visual language models (VLMs), such as LLaVA, Flamingo, or CLIP, have demonstrated impressive performance on various visio-linguistic tasks. Consequently, there are enormous applications of large models that could be potentially used in the biomedical imaging field. Along that direction, there is a lack of related work to show the ability of large models to diagnose the diseases. In this work, we study the zero-shot and few-shot robustness of VLMs on the medical imaging analysis tasks. Our comprehensive experiments demonstrate the effectiveness of VLMs in analyzing biomedical images such as brain MRIs, microscopic images of blood cells, and chest X-rays.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 26 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

15 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V29 Oct

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  12. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
2 more
  1. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  2. 2020COVID-19 Image Data Collection: Prospective Predictions Are the Futurecited by 100

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 14; 2026: 7.

28 papers.

Sort:

By area:

  1. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  2. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  3. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  4. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  5. 2024Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysiscited by 7
  6. 2024MedPix 2.0: A Comprehensive Multimodal Biomedical Data set for Advanced AI Applications with Retrieval Augmented Generation and Knowledge Graphscited by 6
  7. 2025AstroLLaVA: towards the unification of astronomical data and natural languagecited by 6
  8. 2025Trustworthy Medical Imaging with Large Language Models: A Study of Hallucinations Across Modalitiescited by 6
  9. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 6
  10. 2024Efficient and Accurate Pneumonia Detection Using a Novel Multi-Scale Transformer Approachcited by 5
  11. 2025CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classificationcited by 4
  12. 2025Robust Fairness Vision-Language Learning for Medical Image Analysiscited by 4
16 more
  1. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probingcited by 3
  2. 2024Leveraging Expert Input for Robust and Explainable AI-Assisted Lung Cancer Detection in Chest X-rayscited by 2
  3. 2025Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AIcited by 2
  4. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Modelscited by 2
  5. 2026LLM is Not All You Need: A Systematic Evaluation of ML vs. Foundation Models for text and image based Medical Classificationcited by 2
  6. 2024Leveraging Multimodal Models for Enhanced Neuroimaging Diagnostics in Alzheimer's Diseasecited by 1
  7. 2025Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcarecited by 1
  8. 2025Can Large Language Models Challenge CNNs in Medical Image Analysis?cited by 1
  9. 2025Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Modelscited by 1
  10. 2025Neurosymbolic Inference On Foundation Models For Remote Sensing Text-to-image Retrieval With Complex Queriescited by 1
  11. 2026Medical Context Distorts Decisions in Clinical Vision Language Modelscited by 1
  12. 2025Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languagescited by 0
  13. 2026TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuningcited by 0
  14. 2026Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generationcited by 0
  15. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attackscited by 0
  16. 2026What's the Catch? Evaluating Temporal Consistency in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  2. 2025Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis
  3. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  4. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  5. 2025MedM-VL: What Makes a Good Medical LVLM?
  6. 2023Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime
  7. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
  8. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound