# 3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks

arXiv: https://arxiv.org/abs/2506.11147 · HTML: https://mlsanity.com/paper/2506.11147/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [On the Challenges and Perspectives of Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2306.05705/) (2023)
- [CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging](https://mlsanity.com/paper/2403.06801/) (2024)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [BMAD: Benchmarks for Medical Anomaly Detection](https://mlsanity.com/paper/2306.11876/) (2023)
- [Overcoming Data Limitation in Medical Visual Question Answering](https://mlsanity.com/paper/1909.11867/) (2019)
- [Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images](https://mlsanity.com/paper/2403.12570/) (2024)
- [Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models](https://mlsanity.com/paper/2303.05977/) (2023)
- [Multiple Meta-model Quantifying for Medical Visual Question Answering](https://mlsanity.com/paper/2105.08913/) (2021)
- [Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models](https://mlsanity.com/paper/2404.10237/) (2024)
- [MedCoT: Medical Chain of Thought via Hierarchical Expert](https://mlsanity.com/paper/2412.13736/) (2024)
- [A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis](https://mlsanity.com/paper/2307.01981/) (2023)
- [Deep Lesion Tracker: Monitoring Lesions in 4D Longitudinal Imaging Studies](https://mlsanity.com/paper/2012.04872/) (2020)
- [R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest](https://mlsanity.com/paper/2410.20327/) (2024)
- [OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding](https://mlsanity.com/paper/2504.14692/) (2025)
- [KPL: Training-Free Medical Knowledge Mining of Vision-Language Models](https://mlsanity.com/paper/2501.11231/) (2025)
- [Longitudinal Segmentation of MS Lesions via Temporal Difference Weighting](https://mlsanity.com/paper/2409.13416/) (2024)
- [Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Models](https://mlsanity.com/paper/2502.06094/) (2025)

## Built on by (most cited first, 26 of 26)

- [OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis](https://mlsanity.com/paper/2602.16110/) (2026)
- [RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering](https://mlsanity.com/paper/2512.17396/) (2025)
- [Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2603.25155/) (2026)
- [3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis](https://mlsanity.com/paper/2602.18064/) (2026)
- [PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting](https://mlsanity.com/paper/2510.27680/) (2025)
- [RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology](https://mlsanity.com/paper/2605.10761/) (2026)
- [MedReason-R1: Learning to Reason for CT Diagnosis with Reinforcement Learning and Local Zoom](https://mlsanity.com/paper/2510.19626/) (2025)
- [MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis](https://mlsanity.com/paper/2602.22955/) (2026)
- [TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis](https://mlsanity.com/paper/2603.05867/) (2026)
- [Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space](https://mlsanity.com/paper/2603.13800/) (2026)
- [ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding](https://mlsanity.com/paper/2607.24743/) (2026)
- [Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysis](https://mlsanity.com/paper/2603.19516/) (2026)
- [OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation](https://mlsanity.com/paper/2606.17536/) (2026)
- [MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suites](https://mlsanity.com/paper/2510.12126/) (2025)
- [MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models](https://mlsanity.com/paper/2603.11625/) (2026)
- [SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning](https://mlsanity.com/paper/2604.09887/) (2026)
- [Astra: a generalizable report generation foundation model for 3D computed tomography](https://mlsanity.com/paper/2605.31437/) (2026)
- [A Vision-language Framework for Comparative Reasoning in Radiology](https://mlsanity.com/paper/2606.06407/) (2026)
- [RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understanding](https://mlsanity.com/paper/2607.22293/) (2026)
- [NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding](https://mlsanity.com/paper/2605.20525/) (2026)
- [SPARC-Rad: A Multimodal Benchmark Dataset and Evaluation Pipeline for Spatial and Anatomical Reasoning in Radiology Vision-Language Models](https://mlsanity.com/paper/2608.00100/) (2026)
- [CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding](https://mlsanity.com/paper/2608.04515/) (2026)
- [CT-DeltaBench: A Benchmark for Longitudinal 3D Medical Imaging Difference Reporting with Vision-Language Models](https://mlsanity.com/paper/2608.11534/) (2026)
- [HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulation](https://mlsanity.com/paper/2608.12904/) (2026)
- [SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering](https://mlsanity.com/paper/2608.25630/) (2026)
- [BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning](https://mlsanity.com/paper/2610.06571/) (2026)

## Abstract (arXiv)

Medical Visual Question Answering (Med-VQA) holds significant potential for clinical decision support, yet existing efforts primarily focus on 2D imaging with limited task diversity. This paper presents 3D-RAD, a large-scale dataset designed to advance 3D Med-VQA using radiology CT scans. The 3D-RAD dataset encompasses six diverse VQA tasks: anomaly detection, image observation, medical computation, existence detection, static temporal diagnosis, and longitudinal temporal diagnosis. It supports both open- and closed-ended questions while introducing complex reasoning challenges, including computational tasks and multi-stage temporal analysis, to enable comprehensive benchmarking. Extensive evaluations demonstrate that existing vision-language models (VLMs), especially medical VLMs exhibit limited generalization, particularly in multi-temporal tasks, underscoring the challenges of real-world 3D diagnostic reasoning. To drive future advancements, we release a high-quality training set 3D-RAD-T of 136,195 expert-aligned samples, showing that fine-tuning on this dataset could significantly enhance model performance. Our dataset and code, aiming to catalyze multimodal medical AI research and establish a robust foundation for 3D medical visual understanding, are publicly available at https://github.com/Tang-xiaoxiao/3D-RAD.