mlsanityData to 14 Oct 2026

R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest

arXiv 2410.20327 ↗First posted 27 Oct 2024cs.CV cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence has made significant strides in medical visual question answering (Med-VQA), yet prevalent studies often interpret images holistically, overlooking the visual regions of interest that may contain crucial information, potentially aligning with a doctor's prior knowledge that can be incorporated with minimal annotations (e.g., bounding boxes). To address this gap, this paper introduces R-LLaVA, designed to enhance biomedical VQA understanding by integrating simple medical annotations as prior knowledge directly into the image space through CLIP. These annotated visual regions of interest are then fed into the LLaVA model during training, aiming to enrich the model's understanding of biomedical queries. Experimental evaluation on four standard Med-VQA datasets demonstrates R-LLaVA's superiority over existing state-of-the-art (SoTA) methods. Additionally, to verify the model's capability in visual comprehension, a novel multiple-choice medical visual understanding dataset is introduced, confirming the positive impact of focusing on visual regions of interest in advancing biomedical VQA understanding.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 34 of its 59 references that are in the map. First posted on arXiv 27 Oct 2024.

Builds on

34 of 59 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2024Mixtral of Expertscited by 1,823
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
22 more
  1. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  2. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  3. 2018Bilinear Attention Networkscited by 280
  4. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  5. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  6. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  7. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  8. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  9. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  10. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  11. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  12. 2022All You May Need for VQA are Image Captionscited by 65
  13. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  14. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  15. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32
  16. 2023Q2ATransformer: Improving Medical VQA via an Answer Querying Decodercited by 31
  17. 2023A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosiscited by 31
  18. 2022Self-supervised vision-language pretraining for Medical visual question answeringcited by 30
  19. 2021Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answeringcited by 26
  20. 2020WeaQA: Weak Supervision via Captions for Visual Question Answeringcited by 25
  21. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
  22. 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationalecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 8.

21 papers.

Sort:

By area:

  1. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Taskscited by 26
  2. 2025Think Twice to See More: Iterative Visual Reasoning in Medical VLMscited by 13
  3. 2025Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopycited by 11
  4. 2025KPL: Training-Free Medical Knowledge Mining of Vision-Language Modelscited by 10
  5. 2025Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoningcited by 10
  6. 2025Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Datasetcited by 7
  7. 2026CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Frameworkcited by 4
  8. 2025RAU: Reference-based Anatomical Understanding with Vision Language Modelscited by 3
  9. 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQAcited by 3
  10. 2026Graph-of-Mark: Promote Spatial Reasoning in Multimodal Language Models with Graph-Based Visual Promptingcited by 3
  11. 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Imagescited by 2
  12. 2025Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMscited by 2
9 more
  1. 2026SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?cited by 2
  2. 2025CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Trainingcited by 1
  3. 2026SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learningcited by 1
  4. 2026Improving Medical VQA through Trajectory-Aware Process Supervisioncited by 1
  5. 2025PediatricsMQA: a Multi-modal Pediatrics Question Answering Benchmarkcited by 0
  6. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditionscited by 0
  7. 2026MedP-CLIP: Medical CLIP with Region-Aware Prompt Integrationcited by 0
  8. 2026BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogrammingcited by 0
  9. 2026Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learningcited by 0