mlsanityData to 14 Oct 2026

A Lightweight Large Vision-language Model for Multimodal Medical Images

arXiv 2504.05575 ↗First posted 8 Apr 2025cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical Visual Question Answering (VQA) enhances clinical decision-making by enabling systems to interpret medical images and answer clinical queries. However, developing efficient, high-performance VQA models is challenging due to the complexity of medical imagery and diverse modalities. In this paper, we introduce a lightweight, multimodal VQA model integrating BiomedCLIP for image feature extraction and LLaMA-3 for text processing. Designed for medical VQA tasks, our model achieves state-of-the-art performance on the OmniMedVQA dataset. With approximately 8 billion parameters, it requires only two NVIDIA 40 GB A100 GPUs, demonstrating superior efficiency over larger models. Our results show 73.4% accuracy for open-end questions, surpassing existing models and validating its potential for real-world medical applications. Key contributions include a specialized multimodal VQA model, a resource-efficient architecture, and strong performance in answering open-ended clinical questions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 27 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

17 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
  2. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  7. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,186
  8. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  9. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  10. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  11. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  12. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
2 more
  1. 2017Deep Learning for Medical Image Analysiscited by 401
  2. 2023Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answeringcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
  2. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  3. 2026MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA
  4. 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
  5. 2025Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs
  6. 2025Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG
  7. 2023Visual Question Answering in the Medical Domain
  8. 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationale