# Lesion Harvester: Iteratively Mining Unlabeled Lesions and Hard-Negative Examples at Scale

arXiv: https://arxiv.org/abs/2001.07776 · HTML: https://mlsanity.com/paper/2001.07776/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2020.

## Built on by (most cited first, 11 of 11)

- [Universal Lesion Detection in CT Scans using Neural Network Ensembles](https://mlsanity.com/paper/2111.04886/) (2021)
- [AI-Based Detection, Classification and Prediction/Prognosis in Medical Imaging: Towards Radiophenomics](https://mlsanity.com/paper/2110.10332/) (2021)
- [Hetero-Modal Learning and Expansive Consistency Constraints for Semi-Supervised Detection from Multi-Sequence Data](https://mlsanity.com/paper/2103.12972/) (2021)
- [A Flexible Three-Dimensional Hetero-phase Computed Tomography Hepatocellular Carcinoma (HCC) Detection Algorithm for Generalizable and Practical HCC Screening](https://mlsanity.com/paper/2108.07492/) (2021)
- [An End-to-End Framework For Universal Lesion Detection With Missing Annotations](https://mlsanity.com/paper/2303.15024/) (2023)
- [Tackling the Incomplete Annotation Issue in Universal Lesion Detection Task By Exploratory Training](https://mlsanity.com/paper/2309.13306/) (2023)
- [Class Imbalance Correction for Improved Universal Lesion Detection and Tagging in CT](https://mlsanity.com/paper/2504.05591/) (2025)
- [3D Universal Lesion Detection and Tagging in CT with Self-Training](https://mlsanity.com/paper/2504.05201/) (2025)
- [Correcting Class Imbalances with Self-Training for Improved Universal Lesion Detection and Tagging](https://mlsanity.com/paper/2504.05207/) (2025)
- [MoMBS: Mixed-order minibatch sampling enhances model training from diverse-quality images](https://mlsanity.com/paper/2505.18741/) (2025)
- [A Unified 2D Framework for DeepLesion Detection, Segmentation and Short Report Generation](https://mlsanity.com/paper/2608.02805/) (2026)

## Abstract (arXiv)

Acquiring large-scale medical image data, necessary for training machine learning algorithms, is frequently intractable, due to prohibitive expert-driven annotation costs. Recent datasets extracted from hospital archives, e.g., DeepLesion, have begun to address this problem. However, these are often incompletely or noisily labeled, e.g., DeepLesion leaves over 50% of its lesions unlabeled. Thus, effective methods to harvest missing annotations are critical for continued progress in medical image analysis. This is the goal of our work, where we develop a powerful system to harvest missing lesions from the DeepLesion dataset at high precision. Accepting the need for some degree of expert labor to achieve high fidelity, we exploit a small fully-labeled subset of medical image volumes and use it to intelligently mine annotations from the remainder. To do this, we chain together a highly sensitive lesion proposal generator and a very selective lesion proposal classifier. While our framework is generic, we optimize our performance by proposing a 3D contextual lesion proposal generator and by using a multi-view multi-scale lesion proposal classifier. These produce harvested and hard-negative proposals, which we then re-use to finetune our proposal generator by using a novel hard negative suppression loss, continuing this process until no extra lesions are found. Extensive experimental analysis demonstrates that our method can harvest an additional 9,805 lesions while keeping precision above 90%. To demonstrate the benefits of our approach, we show that lesion detectors trained on our harvested lesions can significantly outperform the same variants only trained on the original annotations, with boost of average precision of 7% to 10%. We open source our annotations at https://github.com/JimmyCai91/DeepLesionAnnotation.