# CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging

arXiv: https://arxiv.org/abs/2403.06801 · HTML: https://mlsanity.com/paper/2403.06801/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Large Memory Layers with Product Keys](https://mlsanity.com/paper/1907.05242/) (2019)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes](https://mlsanity.com/paper/2002.04752/) (2020)
- [Cross-modal Prototype Driven Network for Radiology Report Generation](https://mlsanity.com/paper/2207.04818/) (2022)
- [Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reports](https://mlsanity.com/paper/2306.08749/) (2023)
- [A Systematic Collection of Medical Image Datasets for Deep Learning](https://mlsanity.com/paper/2106.12864/) (2021)
- [GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imaging](https://mlsanity.com/paper/2103.01006/) (2021)
- [Diffusion-Based Hierarchical Multi-Label Object Detection to Analyze Panoramic Dental X-rays](https://mlsanity.com/paper/2303.06500/) (2023)
- [DENTEX: Dental Enumeration and Tooth Pathosis Detection Benchmark for Panoramic X-ray](https://mlsanity.com/paper/2305.19112/) (2023)

## Built on by (most cited first, 30 of 96)

- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [MMedAgent: Learning to Use Medical Tools with Multi-modal Agent](https://mlsanity.com/paper/2407.02483/) (2024)
- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis](https://mlsanity.com/paper/2503.20047/) (2025)
- [Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation](https://mlsanity.com/paper/2502.20056/) (2025)
- [RadGPT: Constructing 3D Image-Text Tumor Datasets](https://mlsanity.com/paper/2501.04678/) (2025)
- [Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning](https://mlsanity.com/paper/2506.02327/) (2025)
- [Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation](https://mlsanity.com/paper/2407.02235/) (2024)
- [Large Language Model with Region-guided Referring and Grounding for CT Report Generation](https://mlsanity.com/paper/2411.15539/) (2024)
- [3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks](https://mlsanity.com/paper/2506.11147/) (2025)
- [CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering](https://mlsanity.com/paper/2505.16229/) (2025)
- [SurGen: Text-Guided Diffusion Model for Surgical Video Generation](https://mlsanity.com/paper/2408.14028/) (2024)
- [Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"](https://mlsanity.com/paper/2409.16603/) (2024)
- [Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging](https://mlsanity.com/paper/2510.20639/) (2025)
- [Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation](https://mlsanity.com/paper/2412.13558/) (2024)
- [3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models](https://mlsanity.com/paper/2409.19330/) (2024)
- [Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model](https://mlsanity.com/paper/2411.12783/) (2024)
- [Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering](https://mlsanity.com/paper/2505.18915/) (2025)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [Learning Segmentation from Radiology Reports](https://mlsanity.com/paper/2507.05582/) (2025)
- [Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation](https://mlsanity.com/paper/2406.07146/) (2024)
- [CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumes](https://mlsanity.com/paper/2408.11965/) (2024)
- [OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis](https://mlsanity.com/paper/2602.16110/) (2026)
- [PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generation](https://mlsanity.com/paper/2508.05353/) (2025)
- [Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks](https://mlsanity.com/paper/2510.14803/) (2025)
- [CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generation](https://mlsanity.com/paper/2508.05375/) (2025)
- [PET2Rep: Towards Vision-Language Model-Drived Automated Radiology Report Generation for Positron Emission Tomography](https://mlsanity.com/paper/2508.04062/) (2025)
- [Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation](https://mlsanity.com/paper/2509.24739/) (2025)
- [CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation](https://mlsanity.com/paper/2505.17167/) (2025)
- [Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2603.25155/) (2026)

## Abstract (arXiv)

Medical imaging plays a crucial role in diagnosis, with radiology reports serving as vital documentation. Automating report generation has emerged as a critical need to alleviate the workload of radiologists. While machine learning has facilitated report generation for 2D medical imaging, extending this to 3D has been unexplored due to computational complexity and data scarcity. We introduce the first method to generate radiology reports for 3D medical imaging, specifically targeting chest CT volumes. Given the absence of comparable methods, we establish a baseline using an advanced 3D vision encoder in medical imaging to demonstrate our method's effectiveness, which leverages a novel auto-regressive causal transformer. Furthermore, recognizing the benefits of leveraging information from previous visits, we augment CT2Rep with a cross-attention-based multi-modal fusion module and hierarchical memory, enabling the incorporation of longitudinal multimodal data. Access our code at https://github.com/ibrahimethemhamamci/CT2Rep