Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
On the map: Retrieval-augmented generation → Medical and biomedical RAG
Abstract (arXiv)
Medical Visual Question Answering (MedVQA) enables natural language queries over medical images to support clinical decision-making and patient care. The MEDIQA-WV 2025 shared task addressed wound-care VQA, requiring systems to generate free-text responses and structured wound attributes from images and patient queries. We present the MasonNLP system, which employs a general-domain, instruction-tuned large language model with a retrieval-augmented generation (RAG) framework that incorporates textual and visual examples from in-domain data. This approach grounds outputs in clinically relevant exemplars, improving reasoning, schema adherence, and response quality across dBLEU, ROUGE, BERTScore, and LLM-based metrics. Our best-performing system ranked 3rd among 19 teams and 51 submissions with an average score of 41.37%, demonstrating that lightweight RAG with general-purpose LLMs -- a minimal inference-time layer that adds a few relevant exemplars via simple indexing and fusion, with no extra training or complex re-ranking -- provides a simple and effective baseline for multimodal clinical NLP tasks.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 20 of its 39 references that are in the map. First posted on arXiv 12 Oct 2025.
Builds on
20 of 39 references.
Closest ancestors · 2
Same direction, newest first
- 2024Benchmarking Retrieval-Augmented Generation for Medicine20 Feb
- 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training1 Mar
Other references · 18
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2015VQA: Visual Question Answeringcited by 2,506
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
- 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
- 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
- 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
6 more
- 2021Medical Visual Question Answering: A Surveycited by 93
- 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
- 2015Ask Your Neurons: A Neural-based Approach to Answering Questions about Imagescited by 51
- 2025MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoningcited by 34
- 2015deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targetscited by 21
- 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQAcited by 7
- 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
- 2026MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graphcited by 0
- 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy13 Jul 2026
- 2026PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA22 May 2026
- 2026MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graph11 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering
- 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
- 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationale
- 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
- 2025Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs
- 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images
- 2025A Lightweight Large Vision-language Model for Multimodal Medical Images
- 2025Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework