Volumetric Radiology AI in the Era of Multimodal Large Language Models
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Advances in multimodal large language models (MLLMs) are extending radiological artificial intelligence (AI) beyond task-specific image analysis toward multimodal understanding and reasoning. Volumetric radiology, however, presents a fundamental representational mismatch: clinical interpretation often requires full-volume spatial context and acquisition-dependent quantitative information, whereas current MLLMs are commonly conditioned on selected two-dimensional (2D) images, compressed visual representations, or report-derived text. Reliable volumetric radiology AI therefore requires representations that preserve task-relevant three-dimensional (3D) information and systems that can access, verify, and integrate this information across clinical workflows. In this Review, we examine more than 200 publications through July 2026. We organize the literature around volumetric representation and multimodal understanding at the model level, agentic orchestration at the system level, and their links to clinical applications and evaluation. We review volumetric foundation models, language alignment and compression strategies, and agentic systems that extend MLLMs through planning, tools, memory, and workflow interaction. We distinguish settings in which selected 2D views or report-mediated reasoning may suffice from those that warrant native volumetric modeling. We also introduce a Claim-Design-Validation framework to assess whether technical, workflow, and clinical claims are matched by appropriate design and validation. Across the literature, native volumetric modeling and agentic capabilities depend on the spatial, quantitative, contextual, and workflow requirements of the intended task. Clinical credibility requires faithful volumetric representation, traceable system behavior, claim-aligned validation, and clearly defined human oversight in realistic workflows.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 134 of its 293 references that are in the map. First posted on arXiv 20 Aug 2026.
Builds on
134 of 293 references.
Closest ancestors · 49
Same direction, newest first
- 2026ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding27 Jul
- 2026RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understanding24 Jul
- 2026CORTEX: A Structured Reasoning Benchmark for Trustworthy 3D Chest CT MLLMs25 Jun
- 2026Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography24 Jun
- 2026E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis22 Jun
- 2026Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priors15 Jun
- 2026GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations2 Jun
- 2026Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging1 Jun
- 2026ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training30 May
- 2026Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation29 May
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis19 May
- 2026Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models9 May
37 more
- 2026MedScribe: Clinically Grounded CT Reporting through Agentic Workflows3 May
- 2026GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI25 Apr
- 2026Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI17 Apr
- 2026RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography16 Apr
- 2026Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis11 Apr
- 2026MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning9 Apr
- 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models26 Mar
- 2026Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space14 Mar
- 2026LoV3D: Grounding Cognitive Prognosis Reasoning in Longitudinal 3D Brain MRI via Regional Volume Assessments12 Mar
- 2026MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models12 Mar
- 2026Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT2 Mar
- 2026U-VLM: Hierarchical Vision Language Modeling for Report Generation28 Feb
- 2026Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D25 Feb
- 2026CT-Flow: Orchestrating CT Interpretation Workflow with Model Context Protocol Servers23 Feb
- 2026Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosis1 Feb
- 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation14 Jan
- 2026MEDVISTAGYM: A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning12 Jan
- 2025SCALE-VLP: Soft-Weighted Contrastive Volumetric Vision-Language Pre-training with Spatial-Knowledge Semantics4 Nov
- 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs2 Nov
- 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging23 Oct
- 2025Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRI30 Sep
- 2025MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting4 Sep
- 2025Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generation29 Jun
- 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning8 Jun
- 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training31 May
- 2025Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering25 May
- 2025NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI20 May
- 2025AutoRad-Lung: A Radiomic-Guided Prompting Autoregressive Vision-Language Model for Lung Nodule Malignancy Prediction26 Mar
- 2025Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding24 Jan
- 2024Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation18 Dec
- 2024UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities13 Dec
- 2024MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training8 Dec
- 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model18 Oct
- 20243D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models28 Sep
- 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation2 Jul
- 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
- 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare27 Oct
Other references · 85
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2023PaLM 2 Technical Reportcited by 1,154
- 2024Phi-4 Technical Reportcited by 675
- 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
- 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
- 2025MedGemma Technical Reportcited by 398
- 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
38 more
- 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 282
- 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 100
- 2025Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understandingcited by 96
- 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
- 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
- 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 68
- 2024TotalSegmentator MRI: Robust Sequence-independent Segmentation of Multiple Anatomic Structures in MRIcited by 58
- 2026MedGemma 1.5 Technical Reportcited by 43
- 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
- 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 25
- 2024AutoRG-Brain: Grounded Report Generation for Brain MRIcited by 23
- 2026Qwen-AgentWorld: Language World Models for General Agentscited by 22
- 2025CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answeringcited by 20
- 2025Towards Scalable Language-Image Pre-training for 3D Medical Imagingcited by 13
- 2025Towards Conversational AI for Disease Managementcited by 11
- 20253D Foundation Model for Generalizable Disease Detection in Head Computed Tomographycited by 10
- 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoningcited by 10
- 2026IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentationcited by 10
- 2024How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?cited by 9
- 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 9
- 2025Towards Universal Text-driven CT Image Segmentationcited by 7
- 2024VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysiscited by 6
- 2025Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformerscited by 6
- 2026MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learningcited by 6
- 2026Curia-2: Scaling Self-Supervised Learning for Radiology Foundation Modelscited by 6
- 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5
- 2025CLARITY: Medical World Model for Guiding Treatment Decisions by Simulating Context-Aware Disease Trajectoriescited by 5
- 20263DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysiscited by 5
- 2026Brain-WM: Brain Glioblastoma World Modelcited by 5
- 2026MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflowscited by 5
- 2026Safety and accuracy follow different scaling laws in clinical large language modelscited by 5
- 2025Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agentcited by 4
- 2026Agentic Large Language Models for Training-Free Neuro-Radiological Image Analysiscited by 4
- 2026NeuroClaw Technical Reportcited by 4
- 2026Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memorycited by 4
- 2026Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learningcited by 4
- 2025PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reportingcited by 3
- 2025Metadata-Aligned 3D MRI Representations for Contrast Understanding and Quality Controlcited by 3
50 most cited of 85 All 85 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
- 2026Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models
- 2025Exploring the Design Space of 3D MLLMs for CT Report Generation
- 2023RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance
- 2025A Multimodal Multi-Agent Framework for Radiology Report Generation
- 2026RadDiff: Describing Differences in Radiology Image Sets with Natural Language
- 2026OralMLLM-Bench: Evaluating Cognitive Capabilities of Multimodal Large Language Models in Dental Practice
- 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement