mlsanityData to 14 Oct 2026

MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering

arXiv 2506.22900 ↗First posted 28 Jun 2025cs.CV cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Medical visual question answering (MedVQA) plays a vital role in clinical decision-making by providing contextually rich answers to image-based queries. Although vision-language models (VLMs) are widely used for this task, they often generate factually incorrect answers. Retrieval-augmented generation addresses this challenge by providing information from external sources, but risks retrieving irrelevant context, which can degrade the reasoning capabilities of VLMs. Re-ranking retrievals, as introduced in existing approaches, enhances retrieval relevance by focusing on query-text alignment. However, these approaches neglect the visual or multimodal context, which is particularly crucial for medical diagnosis. We propose MOTOR, a novel multimodal retrieval and re-ranking approach that leverages grounded captions and optimal transport. It captures the underlying relationships between the query and the retrieved context based on textual and visual information. Consequently, our approach identifies more clinically relevant contexts to augment the VLM input. Empirical analysis and human expert evaluation demonstrate that MOTOR achieves higher accuracy on MedVQA datasets, outperforming state-of-the-art methods by an average of 6.45%. Code is available at https://github.com/BioMedIA-MBZUAI/MOTOR.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 28 Jun 2025.

Builds on

26 of 35 references.

References · 26

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  8. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  10. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  11. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
14 more
  1. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  2. 2024MAIRA-2: Grounded Radiology Report Generationcited by 145
  3. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  4. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  5. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  6. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  7. 2021Medical Visual Question Answering: A Surveycited by 93
  8. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  9. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 60
  10. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  11. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
  12. 2024A Survey on Trustworthiness in Foundation Models for Medical Image Analysiscited by 17
  13. 2023The Impact of ChatGPT and LLMs on Medical Imaging Stakeholders: Perspectives and Use Casescited by 7
  14. 2024MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosiscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Taskscited by 1
  2. 2026Dual Causal Inference: Integrating Backdoor Adjustment and Instrumental Variable Learning for Medical VQAcited by 1
  3. 2026FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&Acited by 1
  4. 2026OTCHA: Optimal Transport-driven Confidence-aware Latent Hub Alignment for Multi-View Medical Image Classificationcited by 0
  5. 2026An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationale
  2. 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
  3. 2026MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering
  4. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
  5. 2026Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering
  6. 2025Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG
  7. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
  8. 2024Targeted Visual Prompting for Medical Visual Question Answering