mlsanityData to 7 Oct 2026

Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering

arXiv 2404.16192 ↗First posted 24 Apr 2024cs.CL cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models, while effective in general domains and showing strong performance in diverse multi-modal applications like visual question-answering (VQA), struggle to maintain the same level of effectiveness in more specialized domains, e.g., medical. We propose a medical vision-language model that integrates large vision and language models adapted for the medical domain. This model goes through three stages of parameter-efficient training using three separate biomedical and radiology multi-modal visual and text datasets. The proposed model achieves state-of-the-art performance on the SLAKE 1.0 medical VQA (MedVQA) dataset with an overall accuracy of 87.5% and demonstrates strong performance on another MedVQA dataset, VQA-RAD, achieving an overall accuracy of 73.2%.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 29 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

18 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  2. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts18 May

Other references · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  5. 2017Mixed Precision Trainingcited by 1,135
  6. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  7. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  8. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  9. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  10. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  11. 2021Medical Visual Question Answering: A Surveycited by 93
  12. 2023Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domaincited by 38
4 more
  1. 2022Differentiable Multi-Agent Actor-Critic for Multi-Step Radiology Report Summarizationcited by 13
  2. 2023shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generationcited by 7
  3. 2021Few-Shot Learning of an Interleaved Text Summarization Model by Pretraining with Synthetic Datacited by 4
  4. 2023RadLing: Towards Efficient Radiology Report Understandingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Modelscited by 8
  2. 2024Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Modelscited by 2
  3. 2025VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Modelscited by 2
  4. 2025A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answeringcited by 1
  5. 2026Noise-Aware Visual Representation Learning for Medical Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
  2. 2023Visual Question Answering in the Medical Domain
  3. 2025Medical Large Vision Language Models with Multi-Image Visual Ability
  4. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
  5. 2025Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning
  6. 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
  7. 2026Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence
  8. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs