# UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation

arXiv: https://arxiv.org/abs/2605.17140 · HTML: https://mlsanity.com/paper/2605.17140/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 69 references that are in the map.
- First posted on arXiv 16 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [DaViT: Dual Attention Vision Transformers](https://mlsanity.com/paper/2204.03645/) (2022)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset](https://mlsanity.com/paper/2406.06512/) (2024)
- [MIRAGE: The Illusion of Visual Understanding](https://mlsanity.com/paper/2603.21687/) (2026)
- [EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images](https://mlsanity.com/paper/2310.18652/) (2023)
- [Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis](https://mlsanity.com/paper/2503.20047/) (2025)
- [The University of California San Francisco Preoperative Diffuse Glioma MRI (UCSF-PDGM) Dataset](https://mlsanity.com/paper/2109.00356/) (2021)
- [Pillar-0: A New Frontier for Radiology Foundation Models](https://mlsanity.com/paper/2511.17803/) (2025)
- [Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training](https://mlsanity.com/paper/2305.07920/) (2023)
- [DCFormer: Efficient 3D Vision-Language Modeling with Decomposed Convolutions](https://mlsanity.com/paper/2502.05091/) (2025)
- [Learning neuroimaging models from health system-scale data](https://mlsanity.com/paper/2509.18638/) (2025)
- [Atlas: Multi-Scale Attention Improves Long Context Image Modeling](https://mlsanity.com/paper/2503.12355/) (2025)
- [Barriers in Integrating Medical Visual Question Answering into Radiology Workflows: A Scoping Review and Clinicians' Insights](https://mlsanity.com/paper/2507.08036/) (2025)

## Abstract (arXiv)

Brain tumor diagnosis is largely dependent on Magnetic Resonance Imaging (MRI) evaluation, which requires radiologists to synthesize thousands of images across multiple 3D sequences and longitudinal studies. This process requires advanced neuro-radiology training, poses substantial cognitive load, and is highly time-consuming. Despite increasing demands in radiology, this expertise is difficult to scale, straining the current health systems. Vision-Language Models (VLMs) provide an opportunity to reduce this burden through a semi-automated, interactive interpretation of complex brain MRIs. However, they are currently underutilized in neuro-oncology due to a lack of specialized benchmarks for evaluating them. We introduce a clinically relevant visual question answering (VQA) benchmark -- the UCSF-PDGM-VQA dataset -- consisting of 2,387 QA pairs from 473 glioma-related MRI studies in the public UCSF-PDGM dataset. We further establish a performance baseline for six state-of-the-art vision-language models (VLMs) and one large language model on this dataset. We find that current models are incapable of effectively processing multi-sequence, 3-dimensional MRI scans, thus resulting in a suppression of visual features and over-reliance on language priors, causing modality collapse. These findings underscore a critical deficiency in current model reliability and safety within clinical settings, necessitating the development of robust, domain-specific VLMs.