# Learning Segmentation from Radiology Reports

arXiv: https://arxiv.org/abs/2507.05582 · HTML: https://mlsanity.com/paper/2507.05582/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](https://mlsanity.com/paper/2404.09556/) (2024)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis](https://mlsanity.com/paper/1908.06912/) (2019)
- [Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset](https://mlsanity.com/paper/2406.06512/) (2024)
- [CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging](https://mlsanity.com/paper/2403.06801/) (2024)
- [AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking](https://mlsanity.com/paper/2407.16697/) (2024)
- [A Data-scalable Transformer for Medical Image Segmentation: Architecture, Model Efficiency, and Benchmark](https://mlsanity.com/paper/2203.00131/) (2022)
- [RadGPT: Constructing 3D Image-Text Tumor Datasets](https://mlsanity.com/paper/2501.04678/) (2025)
- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [Automatic Organ and Pan-cancer Segmentation in Abdomen CT: the FLARE 2023 Challenge](https://mlsanity.com/paper/2408.12534/) (2024)
- [Towards Unifying Anatomy Segmentation: Automated Generation of a Full-body CT Dataset via Knowledge Aggregation and Anatomical Guidelines](https://mlsanity.com/paper/2307.13375/) (2023)
- [PanTS: The Pancreatic Tumor Segmentation Dataset](https://mlsanity.com/paper/2507.01291/) (2025)
- [Annotation-efficient cancer detection with report-guided lesion annotation for deep learning-based prostate cancer detection in bpMRI](https://mlsanity.com/paper/2112.05151/) (2021)
- [Acquiring Weak Annotations for Tumor Localization in Temporal and Volumetric Data](https://mlsanity.com/paper/2310.15098/) (2023)
- [Embracing Massive Medical Data](https://mlsanity.com/paper/2407.04687/) (2024)
- [Expectation-Maximization as the Engine of Scalable Medical Intelligence](https://mlsanity.com/paper/2501.03410/) (2025)

## Built on by (most cited first, 11 of 11)

- [Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Data](https://mlsanity.com/paper/2510.14831/) (2025)
- [Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks](https://mlsanity.com/paper/2510.14803/) (2025)
- [Early and Prediagnostic Detection of Pancreatic Cancer from Computed Tomography](https://mlsanity.com/paper/2601.22134/) (2026)
- [RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology](https://mlsanity.com/paper/2605.10761/) (2026)
- [GLeVE: Graph-Guided Lesion Grounding with Proposal Verification in 3D CT](https://mlsanity.com/paper/2605.22619/) (2026)
- [Multimodal MRI Report Findings Supervised Brain Lesion Segmentation with Substructures](https://mlsanity.com/paper/2602.20994/) (2026)
- [MedConcept: Unsupervised Concept Discovery for Interpretability in Medical VLMs](https://mlsanity.com/paper/2604.11868/) (2026)
- [ReportMedSAM: Guiding Segmentation Through Radiology Reports](https://mlsanity.com/paper/2607.14116/) (2026)
- [Report Supervision](https://mlsanity.com/paper/2608.27668/) (2026)
- [RT-Super: Learning Tumor Segmentation from Longitudinal Images and Reports](https://mlsanity.com/paper/2609.35637/) (2026)
- [Merlin Plus: A Large-Scale, Multi-Cancer, Image-Mask-Report Dataset](https://mlsanity.com/paper/2609.36436/) (2026)

## Abstract (arXiv)

Tumor segmentation in CT scans is key for diagnosis, surgery, and prognosis, yet segmentation masks are scarce because their creation requires time and expertise. Public abdominal CT datasets have from dozens to a couple thousand tumor masks, but hospitals have hundreds of thousands of tumor CTs with radiology reports. Thus, leveraging reports to improve segmentation is key for scaling. In this paper, we propose a report-supervision loss (R-Super) that converts radiology reports into voxel-wise supervision for tumor segmentation AI. We created a dataset with 6,718 CT-Report pairs (from the UCSF Hospital), and merged it with public CT-Mask datasets (from AbdomenAtlas 2.0). We used our R-Super to train with these masks and reports, and strongly improved tumor segmentation in internal and external validation--F1 Score increased by up to 16% with respect to training with masks only. By leveraging readily available radiology reports to supplement scarce segmentation masks, R-Super strongly improves AI performance both when very few training masks are available (e.g., 50), and when many masks were available (e.g., 1.7K). Project: https://github.com/MrGiovanni/R-Super