# CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering

arXiv: https://arxiv.org/abs/2505.16229 · HTML: https://mlsanity.com/paper/2505.16229/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging](https://mlsanity.com/paper/2403.06801/) (2024)
- [A Survey of LLM-based Agents in Medicine: How far are we from Baymax?](https://mlsanity.com/paper/2502.11211/) (2025)
- [MAIRA-1: A specialised large multimodal model for radiology report generation](https://mlsanity.com/paper/2311.13668/) (2023)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis](https://mlsanity.com/paper/2404.16754/) (2024)
- [PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology](https://mlsanity.com/paper/2502.08916/) (2025)
- [MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Vision](https://mlsanity.com/paper/2308.16139/) (2023)
- [PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging](https://mlsanity.com/paper/2401.02797/) (2024)
- [Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation](https://mlsanity.com/paper/2412.13558/) (2024)
- [3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models](https://mlsanity.com/paper/2409.19330/) (2024)
- [CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumes](https://mlsanity.com/paper/2408.11965/) (2024)
- [RadSAM: Segmenting 3D radiological images with a 2D promptable model](https://mlsanity.com/paper/2504.20837/) (2025)
- [Enhancing Single-Slice Segmentation with 3D-to-2D Unpaired Scan Distillation](https://mlsanity.com/paper/2406.12254/) (2024)
- [ViT3D Alignment of LLaMA3: 3D Medical Image Report Generation](https://mlsanity.com/paper/2410.08588/) (2024)
- [Explaining 3D Computed Tomography Classifiers with Counterfactuals](https://mlsanity.com/paper/2502.07156/) (2025)

## Built on by (most cited first, 21 of 21)

- [Multi-step retrieval and reasoning improves radiology question answering with large language models](https://mlsanity.com/paper/2508.00743/) (2025)
- [CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretation](https://mlsanity.com/paper/2510.21324/) (2025)
- [RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography](https://mlsanity.com/paper/2604.15231/) (2026)
- [MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows](https://mlsanity.com/paper/2603.24649/) (2026)
- [LungNoduleAgent: A Collaborative Multi-Agent System for Precision Diagnosis of Lung Nodules](https://mlsanity.com/paper/2511.21042/) (2025)
- [Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation](https://mlsanity.com/paper/2603.15822/) (2026)
- [DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents](https://mlsanity.com/paper/2605.09679/) (2026)
- [Radiologist Copilot: An Agentic Framework Orchestrating Specialized Tools for Reliable Radiology Reporting](https://mlsanity.com/paper/2512.02814/) (2025)
- [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026)
- [Agentic AI in Healthcare & Medicine: A Seven-Dimensional Taxonomy for Empirical Evaluation of LLM-based Agents](https://mlsanity.com/paper/2602.04813/) (2026)
- [CT-Flow: Orchestrating CT Interpretation Workflow with Model Context Protocol Servers](https://mlsanity.com/paper/2603.00123/) (2026)
- [RADAR: A Multimodal Benchmark for 3D Image-Based Radiology Report Review](https://mlsanity.com/paper/2603.06681/) (2026)
- [DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation](https://mlsanity.com/paper/2603.17718/) (2026)
- [GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI](https://mlsanity.com/paper/2605.00876/) (2026)
- [Policy-Driven CT-Agent: Modeling Phase-Aware Diagnostic Control for Clinically Consistent CT Reasoning](https://mlsanity.com/paper/2607.10748/) (2026)
- [Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation](https://mlsanity.com/paper/2608.08713/) (2026)
- [MKG-CARE: Case-Aware Reasoning with Multimodal Knowledge Graphs for Explainable Medical Image Diagnosis](https://mlsanity.com/paper/2605.22547/) (2026)
- [Towards Reliable Fetal Ultrasound Interpretation with Multi-Agent Collaboration](https://mlsanity.com/paper/2605.25357/) (2026)
- [Volumetric Radiology AI in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2608.20549/) (2026)
- [NV-Reason-CT: 3D Visual Language Model for CT Analysis](https://mlsanity.com/paper/2609.27511/) (2026)
- [ReVision3D: Attribution-Guided Recursive Self-Improvement for 3D Medical Perception](https://mlsanity.com/paper/2609.32984/) (2026)

## Abstract (arXiv)

Computed Tomography (CT) scan, which produces 3D volumetric medical data that can be viewed as hundreds of cross-sectional images (a.k.a. slices), provides detailed anatomical information for diagnosis. For radiologists, creating CT radiology reports is time-consuming and error-prone. A visual question answering (VQA) system that can answer radiologists' questions about some anatomical regions on the CT scan and even automatically generate a radiology report is urgently needed. However, existing VQA systems cannot adequately handle the CT radiology question answering (CTQA) task for: (1) anatomic complexity makes CT images difficult to understand; (2) spatial relationship across hundreds slices is difficult to capture. To address these issues, this paper proposes CT-Agent, a multimodal agentic framework for CTQA. CT-Agent adopts anatomically independent tools to break down the anatomic complexity; furthermore, it efficiently captures the across-slice spatial relationship with a global-local token compression strategy. Experimental results on two 3D chest CT datasets, CT-RATE and RadGenome-ChestCT, verify the superior performance of CT-Agent.