mlsanityData to 7 Oct 2026

Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space

arXiv 2603.13800 ↗First posted 14 Mar 2026cs.CVcited by 3

On the map: LLM reasoning → Clinical and biomedical applications

also in 1 more
Abstract (arXiv)

Visual spatial intelligence is critical for medical image interpretation, yet remains largely unexplored in Multimodal Large Language Models (MLLMs) for 3D imaging. This gap persists due to a systemic lack of datasets featuring structured 3D spatial annotations beyond basic labels. In this study, we introduce an agentic pipeline that autonomously synthesizes spatial visual question-answering (VQA) data by orchestrating computational tools such as volume estimation and bounding boxes extraction with multi-agent collaboration and expert radiologist validation. We present SpatialMed, the first comprehensive benchmark for evaluating 3D spatial intelligence in medical MLLMs, comprising 31,253 question-answer pairs across multiple organs and tumor types. Our evaluations on 24 state-of-the-art MLLMs and extensive analyses reveal that current models lack robust spatial reasoning capabilities for medical imaging.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 32 of its 59 references that are in the map. First posted on arXiv 14 Mar 2026.

Builds on

32 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025PRS-Med: Position Reasoning Segmentation in Medical Imaging17 May
  2. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning26 Feb

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2025Qwen3-VL Technical Reportcited by 2,534
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  9. 2025OpenAI GPT-5 System Cardcited by 910
  10. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  12. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
18 more
  1. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  2. 2025MedGemma Technical Reportcited by 398
  3. 2024InternLM2 Technical Reportcited by 391
  4. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  5. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  6. 2019The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomescited by 168
  7. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 154
  8. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  9. 2023SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million maskscited by 54
  10. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  11. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  12. 2024GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMscited by 33
  13. 2023INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosiscited by 24
  14. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Taskscited by 24
  15. 2025ReXGroundingCT: A 3D Chest CT Dataset for Segmentation of Findings from Free-Text Reportscited by 16
  16. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
  17. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  18. 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answeringcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Modelscited by 0
  2. 2026Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomographycited by 0
  3. 2026Volumetric Radiology AI in the Era of Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models
  2. 2026Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI
  3. 2026From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation
  4. 2024Beyond the Hype: A dispassionate look at vision-language models in medical scenario
  5. 2026SPARC-Rad: A Multimodal Benchmark Dataset and Evaluation Pipeline for Spatial and Anatomical Reasoning in Radiology Vision-Language Models
  6. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
  7. 2026Agentic Large Language Models for Training-Free Neuro-Radiological Image Analysis
  8. 2026A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans