# Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models

arXiv: https://arxiv.org/abs/2605.08787 · HTML: https://mlsanity.com/paper/2605.08787/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 9 May 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding](https://mlsanity.com/paper/2501.14548/) (2025)
- [VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge](https://mlsanity.com/paper/2411.12915/) (2024)
- [Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis](https://mlsanity.com/paper/2503.20047/) (2025)
- [E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model](https://mlsanity.com/paper/2410.14200/) (2024)
- [Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation](https://mlsanity.com/paper/2412.13558/) (2024)
- [BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning](https://mlsanity.com/paper/2408.07440/) (2024)
- [Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering](https://mlsanity.com/paper/2505.18915/) (2025)
- [Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images](https://mlsanity.com/paper/2508.00549/) (2025)
- [Frequency Domain Adversarial Training for Robust Volumetric Medical Segmentation](https://mlsanity.com/paper/2307.07269/) (2023)
- [Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift](https://mlsanity.com/paper/2509.09397/) (2025)
- [Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts?](https://mlsanity.com/paper/2409.02729/) (2024)
- [T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis](https://mlsanity.com/paper/2510.27265/) (2025)

## Built on by (most cited first, 4 of 4)

- [CORTEX: A Structured Reasoning Benchmark for Trustworthy 3D Chest CT MLLMs](https://mlsanity.com/paper/2606.27264/) (2026)
- [RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understanding](https://mlsanity.com/paper/2607.22293/) (2026)
- [Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography](https://mlsanity.com/paper/2607.26196/) (2026)
- [Volumetric Radiology AI in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2608.20549/) (2026)

## Abstract (arXiv)

Recent advances in 3D medical vision-language models have enabled joint reasoning over volumetric images and text, showing strong performance in medical visual question-answering (VQA) and report generation. Despite this progress, it remains unclear whether these models learn spatially grounded anatomy from 3D volumes or rely primarily on learned priors and language correlations. This uncertainty stems from the lack of systematic evaluation of semantic-spatial reasoning in volumetric medical VLMs for clinically reliable decision support. To address this gap, we introduce CT-SpatialVQA, a benchmark designed to evaluate semantic-spatial reasoning in 3D CT data. The benchmark comprises 9077 clinically grounded question-answer (QA) pairs derived directly from 1601 radiology reports and CT volumes, which are validated via a robust LLM-assisted pipeline with a 95% human consensus agreement rate. Our dataset requires explicit anatomical localization, laterality awareness, structural comparison, and 3D inter-structure relational reasoning. We also introduce a standardized evaluation protocol and benchmark eight 3D medical VLMs, finding severe degradation on semantic-spatial reasoning tasks, averaging 34% accuracy and often below random, highlighting the need for deeper integration of volumetric evidence for trustworthy clinical use.