mlsanityData to 7 Oct 2026

OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM

arXiv 2402.09181 ↗First posted 14 Feb 2024eess.IV cs.CVcited by 182

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in various multimodal tasks. However, their potential in the medical domain remains largely unexplored. A significant challenge arises from the scarcity of diverse medical images spanning various modalities and anatomical regions, which is essential in real-world medical applications. To solve this problem, in this paper, we introduce OmniMedVQA, a novel comprehensive medical Visual Question Answering (VQA) benchmark. This benchmark is collected from 73 different medical datasets, including 12 different modalities and covering more than 20 distinct anatomical regions. Importantly, all images in this benchmark are sourced from authentic medical scenarios, ensuring alignment with the requirements of the medical field and suitability for evaluating LVLMs. Through our extensive experiments, we have found that existing LVLMs struggle to address these medical VQA problems effectively. Moreover, what surprises us is that medical-specialized LVLMs even exhibit inferior performance to those general-domain models, calling for a more versatile and robust LVLM in the biomedical field. The evaluation results not only reveal the current limitations of LVLM in understanding real medical images but also highlight our dataset's significance. Our code with dataset are available at https://github.com/OpenGVLab/Multi-Modality-Arena.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 128 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

54 of 128 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
38 more
  1. 2023Instruction Tuning with GPT-4cited by 676
  2. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  6. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  7. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  9. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  10. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  11. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 347
  12. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  13. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 320
  14. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  15. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  16. 2019REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographscited by 226
  17. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  18. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  19. 2020Can AI help in screening Viral and COVID-19 pneumonia?cited by 200
  20. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 197
  21. 2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Datasetcited by 184
  22. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  23. 2020A Patient-Centric Dataset of Images and Metadata for Identifying Melanomas Using Clinical Contextcited by 146
  24. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
  25. 2022ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation datasetcited by 104
  26. 2020COVID-19 Image Data Collection: Prospective Predictions Are the Futurecited by 100
  27. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 86
  28. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  29. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  30. 2021Predicting Axillary Lymph Node Metastasis in Early Breast Cancer Using Deep Learning on Primary Tumor Biopsy Slidescited by 43
  31. 2022Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithmcited by 35
  32. 2023AIROGS: Artificial Intelligence for RObust Glaucoma Screening Challengecited by 35
  33. 2020COVIDGR dataset and COVID-SDNet methodology for predicting COVID-19 based on Chest X-Ray imagescited by 32
  34. 2021End-to-end Prostate Cancer Detection in bpMRI via 3D CNNs: Effects of Attention Mechanisms, Clinical Priori and Decoupled False Positive Reductioncited by 32
  35. 2022ADAM Challenge: Detecting Age-related Macular Degeneration from Fundus Imagescited by 25
  36. 2022OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semanticscited by 23
  37. 2019The impact of patient clinical information on automated skin cancer detectioncited by 21
  38. 2022The Extreme Cardiac MRI Analysis Challenge under Respiratory Motion (CMRxMotion)cited by 12

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 74; 2026: 84.

182 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  3. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  4. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  5. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 172
  6. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  7. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  8. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
  9. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  10. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  11. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  12. 2025LLM Agents for Education: Advances and Applicationscited by 59
38 more
  1. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledgecited by 50
  2. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  3. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  4. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  5. 2025X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domainscited by 30
  6. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 28
  7. 2024Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicinecited by 28
  8. 2025Pillar-0: A New Frontier for Radiology Foundation Modelscited by 24
  9. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AIcited by 23
  10. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 22
  11. 2024STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answeringcited by 21
  12. 2024MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Modelscited by 21
  13. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  14. 2025Redundancy Principles for MLLMs Benchmarkscited by 18
  15. 2024A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Modelscited by 16
  16. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  17. 2025Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysiscited by 16
  18. 2024Show Me What and Where has Changed? Question Answering and Grounding for Remote Sensing Change Detectioncited by 15
  19. 2025Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMscited by 14
  20. 2025Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answeringcited by 14
  21. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  22. 2025SAM2 for Image and Video Segmentation: A Comprehensive Surveycited by 13
  23. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  24. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learningcited by 13
  25. 2025Vision Language Models in Medicinecited by 12
  26. 2025Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Modelscited by 12
  27. 2025Think Twice to See More: Iterative Visual Reasoning in Medical VLMscited by 12
  28. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  29. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  30. 2024FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Taskscited by 11
  31. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  32. 2025FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMscited by 11
  33. 2024Beyond the Hype: A dispassionate look at vision-language models in medical scenariocited by 10
  34. 2025The Illusion of Readiness in Health AIcited by 10
  35. 2025MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progressioncited by 9
  36. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 9
  37. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  38. 2025MedSG-Bench: A Benchmark for Medical Image Sequences Groundingcited by 8

50 most cited of 182 All 182 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  2. 2025Medical Large Vision Language Models with Multi-Image Visual Ability
  3. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
  4. 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
  5. 2024ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments
  6. 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images
  7. 2025Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning
  8. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks