# Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation

arXiv: https://arxiv.org/abs/2406.07146 · HTML: https://mlsanity.com/paper/2406.07146/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 13 of 13)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging](https://mlsanity.com/paper/2403.06801/) (2024)
- [3D Deep Learning on Medical Images: A Review](https://mlsanity.com/paper/2004.00218/) (2020)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis](https://mlsanity.com/paper/2404.16754/) (2024)
- [BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrieval](https://mlsanity.com/paper/2403.15992/) (2024)
- [vox2vec: A Framework for Self-supervised Contrastive Learning of Voxel-level Representations in Medical Images](https://mlsanity.com/paper/2307.14725/) (2023)

## Built on by (most cited first, 10 of 10)

- [How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM](https://mlsanity.com/paper/2504.05786/) (2025)
- [Large Language Model with Region-guided Referring and Grounding for CT Report Generation](https://mlsanity.com/paper/2411.15539/) (2024)
- [E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model](https://mlsanity.com/paper/2410.14200/) (2024)
- [Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosis](https://mlsanity.com/paper/2602.01200/) (2026)
- [MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation](https://mlsanity.com/paper/2508.16674/) (2025)
- [DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation](https://mlsanity.com/paper/2603.17718/) (2026)
- [CORTEX: A Structured Reasoning Benchmark for Trustworthy 3D Chest CT MLLMs](https://mlsanity.com/paper/2606.27264/) (2026)
- [EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models](https://mlsanity.com/paper/2606.06379/) (2026)
- [Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context](https://mlsanity.com/paper/2607.05880/) (2026)
- [MS-Exam-Gen: An Automatic Benchmark Construction Toolkit for Evaluating LLMs on Multiple Sclerosis-MRI Imaging Knowledge](https://mlsanity.com/paper/2610.06170/) (2026)

## Abstract (arXiv)

Automatic radiology report generation holds significant potential to streamline the labor-intensive process of report writing by radiologists, particularly for 3D radiographs such as CT scans. While CT scans are critical for clinical diagnostics, they remain less explored compared to 2D radiographs. To date, there has been no comprehensive benchmark for 3D radiograph report generation (3DRRG), nor sufficient investigation into the optimal training strategies for Vision Language Models (VLMs) in this context, particularly with respect to vision encoder choices, visual token compression, and model scaling. In this work, we make three key contributions. We curate **CT-3DRRG**, the largest **publicly** available 3D CT-report dataset, establishing a robust and diverse benchmark for evaluating VLM performance on 3DRRG. Furthermore, we propose a comprehensive training recipe for building high-performing VLMs for 3DRRG, exploring key factors such as vision encoder pretraining strategies, visual token compression, and the impact of data & model scale. Guided by these findings, we introduce **Argus**, a state-of-the-art family of VLMs that achieve superior performance across different model sizes and input 3D medical image resolutions, efficiently processing high-resolution 3D images up to 512 × 512 × 256[1].