mlsanityData to 7 Oct 2026

RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training

arXiv 2303.00534 ↗First posted 1 Mar 2023cs.CV cs.CLcited by 31

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Vision-and-language multi-modal pretraining and fine-tuning have shown great success in visual question answering (VQA). Compared to general domain VQA, the performance of biomedical VQA suffers from limited data. In this paper, we propose a retrieval-augmented pretrain-and-finetune paradigm named RAMM for biomedical VQA to overcome the data limitation issue. Specifically, we collect a new biomedical dataset named PMCPM which offers patient-based image-text pairs containing diverse patient situations from PubMed. Then, we pretrain the biomedical multi-modal model to learn visual and textual representation for image-text pairs and align these representations with image-text contrastive objective (ITC). Finally, we propose a retrieval-augmented method to better use the limited data. We propose to retrieve similar image-text pairs based on ITC from pretraining datasets and introduce a novel retrieval-attention module to fuse the representation of the image and the question with the retrieved images and texts. Experiments demonstrate that our retrieval-augmented pretrain-and-finetune paradigm obtains state-of-the-art performance on Med-VQA2019, Med-VQA2021, VQARAD, and SLAKE datasets. Further analysis shows that the proposed RAMM and PMCPM can enhance biomedical VQA performance compared with previous resources and methods. We will open-source our dataset, codes, and pretrained model.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 34 of its 49 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

34 of 49 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  12. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
22 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  2. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  3. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  4. 2018Bilinear Attention Networkscited by 277
  5. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  6. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  7. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  8. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  9. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  10. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  11. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  12. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  13. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  14. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  15. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  16. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  17. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  18. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  19. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  20. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  21. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  22. 2021Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answeringcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 15; 2026: 3.

31 papers.

Sort:

By area:

  1. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  3. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  4. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  5. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  6. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  7. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
  8. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  9. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Taskscited by 27
  10. 2024Semantic Alignment for Multimodal Large Language Modelscited by 12
  11. 2024Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Modelscited by 8
  12. 2025A Multimodal Multi-Agent Framework for Radiology Report Generationcited by 8
19 more
  1. 2025OpenEvents V1: Large-Scale Benchmark Dataset for Multimodal Event Groundingcited by 8
  2. 2024Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approachcited by 7
  3. 2025MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answeringcited by 5
  4. 2025CogPlanner: Unveiling the Potential of Agentic Multimodal Retrieval Augmented Generation with Planningcited by 4
  5. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  6. 2025SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controllingcited by 3
  7. 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQAcited by 3
  8. 2023LADER: Log-Augmented DEnse Retrieval for Biomedical Literature Searchcited by 2
  9. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  10. 2025How to make Medical AI Systems safer? Simulating Vulnerabilities, and Threats in Multimodal Medical RAG Systemcited by 2
  11. 2023Multi-domain improves out-of-distribution and data-limited scenarios for medical image analysiscited by 1
  12. 2025A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answeringcited by 1
  13. 2025Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAGcited by 1
  14. 2025Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answeringcited by 1
  15. 2026MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Modelscited by 1
  16. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  17. 2025LVLM-Aware Multimodal Retrieval for RAG-Based Medical Diagnosis with General-Purpose Modelscited by 0
  18. 2026InViC: Intent-aware Visual Cues for Medical Visual Question Answeringcited by 0
  19. 2026Candidate-Expanding Routing with Permutation-Stabilized Experts for Mixed-Format Medical VQAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
  2. 2022Self-supervised vision-language pretraining for Medical visual question answering
  3. 2023Visual Question Answering in the Medical Domain
  4. 2026MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA
  5. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  6. 2025A Lightweight Large Vision-language Model for Multimodal Medical Images
  7. 2026Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering
  8. 2026MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering