mlsanityData to 7 Oct 2026

MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models

arXiv 2410.13085 ↗First posted 16 Oct 2024cs.LG cs.CL cs.CVcited by 84

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 1 more
Abstract (arXiv)

Artificial Intelligence (AI) has demonstrated significant potential in healthcare, particularly in disease diagnosis and treatment planning. Recent progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new possibilities for interactive diagnostic tools. However, these models often suffer from factual hallucination, which can lead to incorrect diagnoses. Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to address these issues. However, the amount of high-quality data and distribution shifts between training data and deployment data limit the application of fine-tuning methods. Although RAG is lightweight and effective, existing RAG-based approaches are not sufficiently general to different medical domains and can potentially cause misalignment issues, both between modalities and between the model and the ground truth. In this paper, we propose a versatile multimodal RAG system, MMed-RAG, designed to enhance the factuality of Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an adaptive retrieved contexts selection method, and a provable RAG-based preference fine-tuning strategy. These innovations make the RAG process sufficiently general and reliable, significantly improving alignment when introducing retrieved contexts. Experimental results across five medical datasets (involving radiology, ophthalmology, pathology) on medical VQA and report generation demonstrate that MMed-RAG can achieve an average improvement of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available in https://github.com/richard-peng-xia/MMed-RAG.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 82 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

64 of 82 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models6 Jul
  2. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
  3. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models10 Jun
  4. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models14 Feb
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
38 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2019Publicly Available Clinical BERT Embeddingscited by 995
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  5. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  7. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  9. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  10. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  11. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  13. 2023Towards Generalist Biomedical AIcited by 286
  14. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  15. 2017On the Automatic Generation of Medical Imaging Reportscited by 215
  16. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  17. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  18. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  19. 2021Medical Visual Question Answering: A Surveycited by 93
  20. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  21. 2024Med42-v2: A Suite of Clinical LLMscited by 89
  22. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 84
  23. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  24. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  25. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  26. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  27. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  28. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Modelscited by 57
  29. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  30. 2022Diagnosing failures of fairness transfer across distribution shift in real-world medical settingscited by 45
  31. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  32. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
  33. 2024OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understandingcited by 31
  34. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28
  35. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  36. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 27
  37. 2024Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generationcited by 23
  38. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 47; 2026: 33.

84 papers.

Sort:

By area:

  1. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
  2. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
  3. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  4. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  5. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  6. 2024VISA: Retrieval Augmented Generation with Visual Source Attributioncited by 28
  7. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  8. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 22
  9. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  10. 2025Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queriescited by 19
  11. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  12. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
38 more
  1. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  2. 2025Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learningcited by 15
  3. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  4. 2025Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledgecited by 12
  5. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  6. 2025Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generationcited by 12
  7. 2025End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoningcited by 10
  8. 2024Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAGcited by 9
  9. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
  10. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  11. 2025A Multimodal Multi-Agent Framework for Radiology Report Generationcited by 8
  12. 2025QuRe: Query-Relevant Retrieval through Hard Negative Sampling in Composed Image Retrievalcited by 8
  13. 2025Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notescited by 7
  14. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  15. 2026Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improvecited by 6
  16. 2025Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generationcited by 5
  17. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generationcited by 5
  18. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  19. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Modelscited by 4
  20. 2025MIRA: A Novel Framework for Fusing Modalities in Medical RAGcited by 4
  21. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  22. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  23. 2026MEDVISTAGYM: A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learningcited by 4
  24. 2026Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generationcited by 4
  25. 2024AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articlescited by 3
  26. 2025HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generationcited by 3
  27. 2025CogDoc: Towards Unified thinking in Documentscited by 3
  28. 2026Do Multimodal RAG Systems Leak Data? A Comprehensive Evaluation of Membership Inference and Image Caption Retrieval Attackscited by 3
  29. 2026CrisiSense-RAG: Crisis Sensing Multimodal Retrieval-Augmented Generation for Rapid Disaster Impact Assessmentcited by 3
  30. 2025MrM: Black-Box Membership Inference Attacks against Multimodal RAG Systemscited by 2
  31. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  32. 2025Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Modelscited by 2
  33. 2025BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channelcited by 2
  34. 2026Neuro-Oracle: A Trajectory-Aware Agentic RAG Framework for Interpretable Epilepsy Surgical Prognosiscited by 2
  35. 2025Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyondcited by 1
  36. 2025Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environmentcited by 1
  37. 2025A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answeringcited by 1
  38. 2025Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAGcited by 1

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reducing Hallucinations of Medical Multimodal Large Language Models with Visual Retrieval-Augmented Generation
  2. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance
  3. 2024Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology
  4. 2026Pursuing Best Industrial Practices for Retrieval-Augmented Generation in the Medical Domain
  5. 2024Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report
  6. 2025Retrieval-Augmented Generation in Medicine: A Scoping Review of Technical Implementations, Clinical Applications, and Ethical Considerations
  7. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications
  8. 2026P-RAG: Prompt-Enhanced Parametric RAG with LoRA and Selective CoT for Biomedical and Multi-Hop QA