# Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images: Learning from Radiology Reports and Label Ontology

arXiv: https://arxiv.org/abs/1904.04661 · HTML: https://mlsanity.com/paper/1904.04661/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2019.

## Built on by (most cited first, 18 of 18)

- [Explainable artificial intelligence (XAI) in deep learning-based medical image analysis](https://mlsanity.com/paper/2107.10912/) (2021)
- [Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Review](https://mlsanity.com/paper/2112.12596/) (2021)
- [Explainable Machine Learning with Prior Knowledge: An Overview](https://mlsanity.com/paper/2105.10172/) (2021)
- [CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans](https://mlsanity.com/paper/2301.12291/) (2023)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [Temporal Context Matters: Enhancing Single Image Prediction with Disease Progression Representations](https://mlsanity.com/paper/2203.01933/) (2022)
- [Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classification](https://mlsanity.com/paper/2403.04024/) (2024)
- [Transformer Lesion Tracker](https://mlsanity.com/paper/2206.06252/) (2022)
- [SAM++: Enhancing Anatomic Matching using Semantic Information and Structural Inference](https://mlsanity.com/paper/2306.13988/) (2023)
- [Quantifying Predictive Uncertainty in Medical Image Analysis with Deep Kernel Learning](https://mlsanity.com/paper/2106.00638/) (2021)
- [ADAPT: Alzheimer Diagnosis through Adaptive Profiling Transformers](https://mlsanity.com/paper/2401.06349/) (2024)
- [Towards Highly Expressive Machine Learning Models of Non-Melanoma Skin Cancer](https://mlsanity.com/paper/2207.05749/) (2022)
- [How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses](https://mlsanity.com/paper/2403.05680/) (2024)
- [Class Imbalance Correction for Improved Universal Lesion Detection and Tagging in CT](https://mlsanity.com/paper/2504.05591/) (2025)
- [3DLAND: 3D Lesion Abdominal Anomaly Localization Dataset](https://mlsanity.com/paper/2602.12820/) (2026)
- [Correcting Class Imbalances with Self-Training for Improved Universal Lesion Detection and Tagging](https://mlsanity.com/paper/2504.05207/) (2025)
- [A Hierarchical Slice Attention Network for Appendicitis Classification in 3D CT Scans](https://mlsanity.com/paper/2506.23209/) (2025)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)

## Abstract (arXiv)

In radiologists' routine work, one major task is to read a medical image, e.g., a CT scan, find significant lesions, and describe them in the radiology report. In this paper, we study the lesion description or annotation problem. Given a lesion image, our aim is to predict a comprehensive set of relevant labels, such as the lesion's body part, type, and attributes, which may assist downstream fine-grained diagnosis. To address this task, we first design a deep learning module to extract relevant semantic labels from the radiology reports associated with the lesion images. With the images and text-mined labels, we propose a lesion annotation network (LesaNet) based on a multilabel convolutional neural network (CNN) to learn all labels holistically. Hierarchical relations and mutually exclusive relations between the labels are leveraged to improve the label prediction accuracy. The relations are utilized in a label expansion strategy and a relational hard example mining algorithm. We also attach a simple score propagation layer on LesaNet to enhance recall and explore implicit relation between labels. Multilabel metric learning is combined with classification to enable interpretable prediction. We evaluated LesaNet on the public DeepLesion dataset, which contains over 32K diverse lesion images. Experiments show that LesaNet can precisely annotate the lesions using an ontology of 171 fine-grained labels with an average AUC of 0.9344.