mlsanityData to 7 Oct 2026

PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering

arXiv 2305.10415 ↗First posted 17 May 2023cs.CVcited by 333

Not on the map. Why?

Cited by 176 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Medical Visual Question Answering (MedVQA) presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery by leveraging artificial intelligence to interpret and answer questions based on medical images. In this study, we reframe the problem of MedVQA as a generation task that naturally follows the human-machine interaction and propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model. We establish a scalable pipeline to construct a large-scale medical visual question-answering dataset, named PMC-VQA, which contains 227k VQA pairs of 149k images that cover various modalities or diseases. We train the proposed model on PMC-VQA and then fine-tune it on multiple public benchmarks, e.g., VQA-RAD, SLAKE, and Image-Clef-2019, significantly outperforming existing MedVQA models in generating relevant, accurate free-form answers. In addition, we propose a test set that has undergone manual verification, which is significantly more challenging, serving to better monitor the development of generative MedVQA methods. To facilitate comprehensive evaluation and comparison, we have maintained a leaderboard at https://paperswithcode.com/paper/pmc-vqa-visual-instruction-tuning-for-medical, offering a centralized resource for tracking progress and benchmarking state-of-the-art approaches. The PMC-VQA dataset emerges as a vital resource for the field of research, and the MedVInT presents a significant breakthrough in the area of MedVQA.

Key facts. Cited by 333 papers in the map (data through 7 Oct 2026). Builds on 29 of its 72 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

29 of 72 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
17 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  4. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  6. 2021The Medical Segmentation Decathloncited by 620
  7. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  8. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  9. 2020CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentationcited by 307
  10. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  11. 2020MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysiscited by 206
  12. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  13. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  14. 2021Medical Visual Question Answering: A Surveycited by 93
  15. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
  16. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  17. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 31; 2024: 76; 2025: 124; 2026: 102.

333 papers.

Sort:

By area:

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  6. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  7. 2024What matters when building vision-language models?cited by 321
  8. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  9. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  10. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  11. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  12. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
38 more
  1. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  2. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  3. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  4. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  5. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  6. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 172
  7. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  8. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  9. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 154
  10. 2026Qwen3.5-Omni Technical Reportcited by 146
  11. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  12. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  13. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  14. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  15. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  16. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  17. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  18. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  19. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  20. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  21. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 84
  22. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  23. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  24. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  25. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  26. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  27. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  28. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
  29. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  30. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  31. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  32. 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 49
  33. 2024MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detectioncited by 46
  34. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  35. 2024PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathologycited by 43
  36. 2023CLIP in Medical Imaging: A Surveycited by 40
  37. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4Vcited by 38
  38. 2025Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detectorcited by 38

50 most cited of 333 All 333 (JSON)