Exploring the Design Space of 3D MLLMs for CT Report Generation
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) have emerged as a promising way to automate Radiology Report Generation (RRG). In this work, we systematically investigate the design space of 3D MLLMs, including visual input representation, projectors, Large Language Models (LLMs), and fine-tuning techniques for 3D CT report generation. We also introduce two knowledge-based report augmentation methods that improve performance on the GREEN score by up to 10%, achieving the 2nd place on the MICCAI 2024 AMOS-MM challenge. Our results on the 1,687 cases from the AMOS-MM dataset show that RRG is largely independent of the size of LLM under the same training protocol. We also show that larger volume size does not always improve performance if the original ViT was pre-trained on a smaller volume size. Lastly, we show that using a segmentation mask along with the CT volume improves performance. The code is publicly available at https://github.com/bowang-lab/AMOS-MM-Solution
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 31 references that are in the map. First posted on arXiv 26 Jun 2025.
Builds on
18 of 31 references.
Closest ancestors · 3
Same direction, newest first
- 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
- 2023CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images22 Oct
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
Other references · 15
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Visual Instruction Tuningcited by 7,869
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
- 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 351
- 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
- 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
3 more
- 2024GREEN: Generative Radiology Report Evaluation and Error Notationcited by 128
- 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
- 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation
- 20243D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models
- 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation
- 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement
- 2026U-VLM: Hierarchical Vision Language Modeling for Report Generation
- 2026Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression
- 2025Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuning
- 2024MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models