# Visual Grounding of Whole Radiology Reports for 3D CT Images arXiv: https://arxiv.org/abs/2312.04794 ยท HTML: https://mlsanity.com/paper/2312.04794/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 19 of its 29 references that are in the map. - First posted on arXiv 8 Dec 2023. ## Builds on (most cited first, 19 of 19) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016) - [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019) - [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019) - [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014) - [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021) - [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018) - [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021) - [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016) - [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019) - [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018) - [Deep Fragment Embeddings for Bidirectional Image Sentence Mapping](https://mlsanity.com/paper/1406.5679/) (2014) - [AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation](https://mlsanity.com/paper/2203.10095/) (2022) - [Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video](https://mlsanity.com/paper/2203.06667/) (2022) - [A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/2009.01523/) (2020) - [Detailed Annotations of Chest X-Rays via CT Projection for Report Understanding](https://mlsanity.com/paper/2210.03416/) (2022) - [Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports](https://mlsanity.com/paper/2110.03094/) (2021) - [Automatic Segmentation, Localization, and Identification of Vertebrae in 3D CT Images Using Cascaded Convolutional Neural Networks](https://mlsanity.com/paper/2009.13798/) (2020) - [Computation of Total Kidney Volume from CT images in Autosomal Dominant Polycystic Kidney Disease using Multi-Task 3D Convolutional Neural Networks](https://mlsanity.com/paper/1809.02268/) (2018) ## Built on by (most cited first, 13 of 13) - [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024) - [ChEX: Interactive Localization and Region Description in Chest X-rays](https://mlsanity.com/paper/2404.15770/) (2024) - [Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding](https://mlsanity.com/paper/2404.06798/) (2024) - [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024) - [TotalFM: An Organ-Separated 3D-CT Foundation Model Leveraging Large-Scale Routine Clinical Radiology Data](https://mlsanity.com/paper/2601.00260/) (2026) - [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025) - [ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue](https://mlsanity.com/paper/2409.17610/) (2024) - [Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models](https://mlsanity.com/paper/2507.12236/) (2025) - [Generalised Medical Phrase Grounding](https://mlsanity.com/paper/2512.01085/) (2025) - [Enhancing Medical Visual Grounding via Knowledge-guided Spatial Prompts](https://mlsanity.com/paper/2604.01915/) (2026) - [GLeVE: Graph-Guided Lesion Grounding with Proposal Verification in 3D CT](https://mlsanity.com/paper/2605.22619/) (2026) - [MonteRET: AI Agent Enhancing Multimodal LLMs with Multi-granularity Knowledge Retrieval for Chest CT Report Generation](https://mlsanity.com/paper/2607.14264/) (2026) - [VICCA: Visual Interpretation and Comprehension of Chest X-ray Anomalies in Generated Report Without Human Feedback](https://mlsanity.com/paper/2501.17726/) (2025) ## Abstract (arXiv) Building a large-scale training dataset is an essential problem in the development of medical image recognition systems. Visual grounding techniques, which automatically associate objects in images with corresponding descriptions, can facilitate labeling of large number of images. However, visual grounding of radiology reports for CT images remains challenging, because so many kinds of anomalies are detectable via CT imaging, and resulting report descriptions are long and complex. In this paper, we present the first visual grounding framework designed for CT image and report pairs covering various body parts and diverse anomaly types. Our framework combines two components of 1) anatomical segmentation of images, and 2) report structuring. The anatomical segmentation provides multiple organ masks of given CT images, and helps the grounding model recognize detailed anatomies. The report structuring helps to accurately extract information regarding the presence, location, and type of each anomaly described in corresponding reports. Given the two additional image/report features, the grounding model can achieve better localization. In the verification process, we constructed a large-scale dataset with region-description correspondence annotations for 10,410 studies of 7,321 unique patients. We evaluated our framework using grounding accuracy, the percentage of correctly localized anomalies, as a metric and demonstrated that the combination of the anatomical segmentation and the report structuring improves the performance with a large margin over the baseline model (66.0% vs 77.8%). Comparison with the prior techniques also showed higher performance of our method.