mlsanityData to 14 Oct 2026

Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering

arXiv 2505.18915 ↗First posted 25 May 2025cs.CVcited by 14

On the map: LLM reasoning → Clinical and biomedical applications

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have shown promise in various 2D visual tasks, yet their readiness for 3D clinical diagnosis remains unclear due to stringent demands for recognition precision, reasoning ability, and domain knowledge. To systematically evaluate these dimensions, we present DeepTumorVQA, a diagnostic visual question answering (VQA) benchmark targeting abdominal tumors in CT scans. It comprises 9,262 CT volumes (3.7M slices) from 17 public datasets, with 395K expert-level questions spanning four categories: Recognition, Measurement, Visual Reasoning, and Medical Reasoning. DeepTumorVQA introduces unique challenges, including small tumor detection and clinical reasoning across 3D anatomy. Benchmarking four advanced VLMs (RadFM, M3D, Merlin, CT-CHAT), we find current models perform adequately on measurement tasks but struggle with lesion recognition and reasoning, and are still not meeting clinical needs. Two key insights emerge: (1) large-scale multimodal pretraining plays a crucial role in DeepTumorVQA testing performance, making RadFM stand out among all VLMs. (2) Our dataset exposes critical differences in VLM components, where proper image preprocessing and design of vision modules significantly affect 3D perception. To facilitate medical multimodal research, we have released DeepTumorVQA as a rigorous benchmark: https://github.com/Schuture/DeepTumorVQA.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 34 of its 56 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

34 of 56 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  9. 2021The Medical Segmentation Decathloncited by 628
  10. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  11. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  12. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
22 more
  1. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 509
  2. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  3. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  4. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 351
  5. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  6. 2023Towards Generalist Biomedical AIcited by 289
  7. 2020AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?cited by 224
  8. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 199
  9. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 184
  10. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  11. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 162
  12. 2015DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentationcited by 159
  13. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  14. 2024Merlin: A Computed Tomography Vision-Language Foundation Model and Datasetcited by 102
  15. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  16. 2021Medical Visual Question Answering: A Surveycited by 93
  17. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  18. 2023GenerateCT: Text-Conditional Generation of 3D Chest CT Volumescited by 64
  19. 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 42
  20. 2025RadGPT: Constructing 3D Image-Text Tumor Datasetscited by 36
  21. 2024AutoRG-Brain: Grounded Report Generation for Brain MRIcited by 24
  22. 2025CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understandingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Maskscited by 9
  2. 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answeringcited by 6
  3. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  4. 20263DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysiscited by 5
  5. 2026Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Modelscited by 4
  6. 2026RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiologycited by 4
  7. 2025See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancementcited by 3
  8. 2026TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysiscited by 3
  9. 2025Auditing Significance, Metric Choice, and Demographic Fairness in Medical AI Challengescited by 2
  10. 2026CAP-IQA: Context-Aware Prompt-Guided CT Image Quality Assessmentcited by 0
  11. 2026CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMscited by 0
  12. 2026Beyond Masks: The Case for Medical Image Parsingcited by 0
2 more
  1. 2026Volumetric Radiology AI in the Era of Multimodal Large Language Modelscited by 0
  2. 2026SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents
  2. 2026Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
  3. 2026UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation
  4. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks
  5. 2025MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports
  6. 2025Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy
  7. 2025MedVision: Benchmarking Quantitative Medical Image Analysis
  8. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding