# Large Language Model with Region-guided Referring and Grounding for CT Report Generation

arXiv: https://arxiv.org/abs/2411.15539 · HTML: https://mlsanity.com/paper/2411.15539/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 23 Nov 2024.

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest](https://mlsanity.com/paper/2307.03601/) (2023)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models](https://mlsanity.com/paper/2404.13013/) (2024)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging](https://mlsanity.com/paper/2403.06801/) (2024)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation](https://mlsanity.com/paper/2308.12604/) (2023)
- [RegionGPT: Towards Region Understanding Vision Language Model](https://mlsanity.com/paper/2403.02330/) (2024)
- [RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis](https://mlsanity.com/paper/2404.16754/) (2024)
- [Dia-LLaMA: Towards Large Language Model-driven CT Report Generation](https://mlsanity.com/paper/2403.16386/) (2024)
- [MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis](https://mlsanity.com/paper/2407.04106/) (2024)
- [Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2410.18387/) (2024)
- [A Refer-and-Ground Multimodal Large Language Model for Biomedicine](https://mlsanity.com/paper/2406.18146/) (2024)

## Built on by (most cited first, 28 of 28)

- [CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generation](https://mlsanity.com/paper/2508.05375/) (2025)
- [PET2Rep: Towards Vision-Language Model-Drived Automated Radiology Report Generation for Positron Emission Tomography](https://mlsanity.com/paper/2508.04062/) (2025)
- [A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generation](https://mlsanity.com/paper/2508.09566/) (2025)
- [Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generation](https://mlsanity.com/paper/2506.23102/) (2025)
- [Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2509.03906/) (2025)
- [MvKeTR: Chest CT Report Generation with Multi-View Perception and Knowledge Enhancement](https://mlsanity.com/paper/2411.18309/) (2024)
- [LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learning](https://mlsanity.com/paper/2506.17562/) (2025)
- [PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting](https://mlsanity.com/paper/2510.27680/) (2025)
- [MARCH: Multi-Agent Radiology Clinical Hierarchy for CT Report Generation](https://mlsanity.com/paper/2604.16175/) (2026)
- [Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidance](https://mlsanity.com/paper/2604.10437/) (2026)
- [Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation](https://mlsanity.com/paper/2605.30984/) (2026)
- [Rethinking Whole-Body CT Image Interpretation: An Abnormality-Centric Approach](https://mlsanity.com/paper/2506.03238/) (2025)
- [Vision-Language Models for Automated 3D PET/CT Report Generation](https://mlsanity.com/paper/2511.20145/) (2025)
- [Radiologist Copilot: An Agentic Framework Orchestrating Specialized Tools for Reliable Radiology Reporting](https://mlsanity.com/paper/2512.02814/) (2025)
- [GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training](https://mlsanity.com/paper/2603.06467/) (2026)
- [DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation](https://mlsanity.com/paper/2603.17718/) (2026)
- [SliceWorld: A Predictive and Controllable World-State Model for CT Report Generation](https://mlsanity.com/paper/2605.24371/) (2026)
- [Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priors](https://mlsanity.com/paper/2606.17213/) (2026)
- [MonteRET: AI Agent Enhancing Multimodal LLMs with Multi-granularity Knowledge Retrieval for Chest CT Report Generation](https://mlsanity.com/paper/2607.14264/) (2026)
- [Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation](https://mlsanity.com/paper/2608.08713/) (2026)
- [RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching](https://mlsanity.com/paper/2609.01470/) (2026)
- [Unleashing Video Language Models for Fine-grained HRCT Report Generation](https://mlsanity.com/paper/2603.12469/) (2026)
- [EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models](https://mlsanity.com/paper/2606.06379/) (2026)
- [DECIS: Dual-Evidence Corrective Verification for Interpretable Strabismus Diagnostic Decision-Making](https://mlsanity.com/paper/2606.09249/) (2026)
- [ReMAP-PET: Beyond Visual Understanding -- Learning Region-Guided Metabolic Alignment Semantics from Brain PET](https://mlsanity.com/paper/2606.29577/) (2026)
- [ALTER: Modeling Longitudinal Changes via Regional Differencing for 3D CT Report Generation](https://mlsanity.com/paper/2608.05615/) (2026)
- [Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation](https://mlsanity.com/paper/2608.08494/) (2026)
- [GateSPINE: Gated Cross-View Fusion for Lumbar Spine MRI Report Generation](https://mlsanity.com/paper/2609.40091/) (2026)

## Abstract (arXiv)

Computed tomography (CT) report generation is crucial to assist radiologists in interpreting CT volumes, which can be time-consuming and labor-intensive. Existing methods primarily only consider the global features of the entire volume, making it struggle to focus on specific regions and potentially missing abnormalities. To address this issue, we propose Reg2RG, the first region-guided referring and grounding framework for CT report generation, which enhances diagnostic performance by focusing on anatomical regions within the volume. Specifically, we utilize masks from a universal segmentation module to capture local features for each referring region. A local feature decoupling (LFD) strategy is proposed to preserve the local high-resolution details with little computational overhead. Then the local features are integrated with global features to capture inter-regional relationships within a cohesive context. Moreover, we propose a novel region-report alignment (RRA) training strategy. It leverages the recognition of referring regions to guide the generation of region-specific reports, enhancing the model's referring and grounding capabilities while also improving the report's interpretability. A large language model (LLM) is further employed as the language decoder to generate reports from integrated visual features, facilitating region-level comprehension. Extensive experiments on two large-scale chest CT-report datasets demonstrate the superiority of our method, which outperforms several state-of-the-art methods in terms of both natural language generation and clinical efficacy metrics while preserving promising interpretability. The code is available at https://github.com/zhi-xuan-chen/Reg2RG.