mlsanityData to 7 Oct 2026

GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis

arXiv 2411.16778 ↗First posted 25 Nov 2024cs.CV cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Medical Visual Question Answering (Med-VQA) combines computer vision and natural language processing to automatically answer clinical inquiries about medical images. However, current Med-VQA datasets exhibit two significant limitations: (1) they often lack visual and textual explanations for answers, hindering comprehension for patients and junior doctors; (2) they typically offer a narrow range of question formats, inadequately reflecting the diverse requirements in practical scenarios. These limitations pose significant challenges to the development of a reliable and user-friendly Med-VQA system. To address these challenges, we introduce a large-scale, Groundable, and Explainable Medical VQA benchmark for chest X-ray diagnosis (GEMeX), featuring several innovative components: (1) a multi-modal explainability mechanism that offers detailed visual and textual explanations for each question-answer pair, thereby enhancing answer comprehensibility; (2) four question types, open-ended, closed-ended, single-choice, and multiple-choice, to better reflect practical needs. With 151,025 images and 1,605,575 questions, GEMeX is the currently largest chest X-ray VQA dataset. Evaluation of 12 representative large vision language models (LVLMs) on GEMeX reveals suboptimal performance, underscoring the dataset's complexity. Meanwhile, we propose a strong model by fine-tuning an existing LVLM on the GEMeX training set. The substantial performance improvement showcases the dataset's effectiveness. The benchmark is available at https://www.med-vqa.com/GEMeX.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

49 of 65 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2023Mistral 7Bcited by 3,065
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
37 more
  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  5. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  6. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  7. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  8. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  9. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  10. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  11. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  12. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  13. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 169
  14. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
  15. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  16. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  17. 2021Medical Visual Question Answering: A Surveycited by 93
  18. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
  19. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  20. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-trainingcited by 81
  21. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 74
  22. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  23. 2023LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generationcited by 57
  24. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Modelscited by 57
  25. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  26. 2018VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questionscited by 50
  27. 2023Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answeringcited by 41
  28. 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 41
  29. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 35
  30. 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
  31. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28
  32. 2021Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answeringcited by 26
  33. 2023Towards LLM-driven Dialogue State Trackingcited by 25
  34. 2023SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Modelcited by 18
  35. 2023EasyGen: Easing Multimodal Generation with BiDiffuser and LLMscited by 9
  36. 2023Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmarkcited by 5
  37. 2024EyeGPT: Ophthalmic Assistant with Large Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 18.

28 papers.

Sort:

By area:

  1. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  2. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  3. 2025Think Twice to See More: Iterative Visual Reasoning in Medical VLMscited by 12
  4. 2025MedSG-Bench: A Benchmark for Medical Image Sequences Groundingcited by 8
  5. 2025GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoningcited by 8
  6. 2025UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Modelscited by 5
  7. 2025CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rayscited by 5
  8. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generationcited by 5
  9. 2025VinDr-CXR-VQA: A Visual Question Answering Dataset for Explainable Chest X-Ray Analysis with Multi-Task Learningcited by 3
  10. 2025MedRepBench: A Comprehensive Benchmark for Medical Report Interpretationcited by 2
  11. 2026A data- and compute-efficient chest X-ray foundation model beyond aggressive scalingcited by 2
  12. 2026Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answeringcited by 2
16 more
  1. 2026Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysiscited by 2
  2. 2026Benchmarking PhD-Level Coding in 3D Geometric Computer Visioncited by 2
  3. 2026MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answeringcited by 1
  4. 2026Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRIcited by 1
  5. 2026Towards Clinically Interpretable Ophthalmic VQA via Spatially-Grounded Lesion Evidencecited by 1
  6. 2026Hallucination Detection and Correction in Medical VLMs via Counter-Evidence Verificationcited by 1
  7. 2026CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rayscited by 1
  8. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
  9. 2026MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQAcited by 1
  10. 2026Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledgecited by 0
  11. 2026LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-raycited by 0
  12. 2026Deterministic Hallucination Detection in Medical VQA via Confidence-Evidence Bayesian Gaincited by 0
  13. 2026NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understandingcited by 0
  14. 2026Solving the Needle-in-a-Haystack Problem in Mammography Vision-Language Model with Differentiable Subset Samplingcited by 0
  15. 2026LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarioscited by 0
  16. 2026UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasoundcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports
  2. 2025MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering
  3. 2023Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning
  4. 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
  5. 2024Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays
  6. 2025Interpreting Chest X-rays Like a Radiologist: A Benchmark with Clinical Reasoning
  7. 2025MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression
  8. 2024WoLF: Wide-scope Large Language Model Framework for CXR Understanding