# Learning from Multiple Datasets with Heterogeneous and Partial Labels for Universal Lesion Detection in CT

arXiv: https://arxiv.org/abs/2009.02577 · HTML: https://mlsanity.com/paper/2009.02577/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2020.

## Built on by (most cited first, 30 of 42)

- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks](https://mlsanity.com/paper/2305.09666/) (2023)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models](https://mlsanity.com/paper/2103.00429/) (2021)
- [SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection](https://mlsanity.com/paper/2412.20665/) (2024)
- [Label-Assemble: Leveraging Multiple Datasets with Partial Labels](https://mlsanity.com/paper/2109.12265/) (2021)
- [Spatial Consistency Loss for Training Multi-Label Classifiers from Single-Label Annotations](https://mlsanity.com/paper/2203.06127/) (2022)
- [SATr: Slice Attention with Transformer for Universal Lesion Detection](https://mlsanity.com/paper/2203.07373/) (2022)
- [Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss](https://mlsanity.com/paper/2105.01218/) (2021)
- [Federated Multi-organ Segmentation with Inconsistent Labels](https://mlsanity.com/paper/2206.07156/) (2022)
- [CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans](https://mlsanity.com/paper/2301.12291/) (2023)
- [SparseDet: Improving Sparsely Annotated Object Detection with Pseudo-positive Mining](https://mlsanity.com/paper/2201.04620/) (2022)
- [Lesion Segmentation and RECIST Diameter Prediction via Click-driven Attention and Dual-path Connection](https://mlsanity.com/paper/2105.01828/) (2021)
- [Universal Lesion Detection in CT Scans using Neural Network Ensembles](https://mlsanity.com/paper/2111.04886/) (2021)
- [Feasibility and benefits of joint learning from MRI databases with different brain diseases and modalities for segmentation](https://mlsanity.com/paper/2405.18511/) (2024)
- [Transformer Lesion Tracker](https://mlsanity.com/paper/2206.06252/) (2022)
- [EEG Decoding for Datasets with Heterogenous Electrode Configurations using Transfer Learning Graph Neural Networks](https://mlsanity.com/paper/2306.13109/) (2023)
- [Conditional Training with Bounding Map for Universal Lesion Detection](https://mlsanity.com/paper/2103.12277/) (2021)
- [SAM++: Enhancing Anatomic Matching using Semantic Information and Structural Inference](https://mlsanity.com/paper/2306.13988/) (2023)
- [A Cascaded Approach for ultraly High Performance Lesion Detection and False Positive Removal in Liver CT Scans](https://mlsanity.com/paper/2306.16036/) (2023)
- [Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalities](https://mlsanity.com/paper/2406.11636/) (2024)
- [Anatomy-Aware Lymph Node Detection in Chest CT using Implicit Station Stratification](https://mlsanity.com/paper/2307.15271/) (2023)
- [Learning from Partially Overlapping Labels: Image Segmentation under Annotation Shift](https://mlsanity.com/paper/2107.05938/) (2021)
- [ConUNETR: A Conditional Transformer Network for 3D Micro-CT Embryonic Cartilage Segmentation](https://mlsanity.com/paper/2402.03695/) (2024)
- [One model to use them all: Training a segmentation model with complementary datasets](https://mlsanity.com/paper/2402.19340/) (2024)
- [Effective Lymph Nodes Detection in CT Scans Using Location Debiased Query Selection and Contrastive Query Representation in Transformer](https://mlsanity.com/paper/2404.03819/) (2024)
- [SGDA: Towards 3D Universal Pulmonary Nodule Detection via Slice Grouped Domain Attention](https://mlsanity.com/paper/2303.03625/) (2023)
- [The Missing Piece: A Case for Pre-Training in 3D Medical Object Detection](https://mlsanity.com/paper/2509.15947/) (2025)
- [Hetero-Modal Learning and Expansive Consistency Constraints for Semi-Supervised Detection from Multi-Sequence Data](https://mlsanity.com/paper/2103.12972/) (2021)

## Abstract (arXiv)

Large-scale datasets with high-quality labels are desired for training accurate deep learning models. However, due to the annotation cost, datasets in medical imaging are often either partially-labeled or small. For example, DeepLesion is such a large-scale CT image dataset with lesions of various types, but it also has many unlabeled lesions (missing annotations). When training a lesion detector on a partially-labeled dataset, the missing annotations will generate incorrect negative signals and degrade the performance. Besides DeepLesion, there are several small single-type datasets, such as LUNA for lung nodules and LiTS for liver tumors. These datasets have heterogeneous label scopes, i.e., different lesion types are labeled in different datasets with other types ignored. In this work, we aim to develop a universal lesion detection algorithm to detect a variety of lesions. The problem of heterogeneous and partial labels is tackled. First, we build a simple yet effective lesion detection framework named Lesion ENSemble (LENS). LENS can efficiently learn from multiple heterogeneous lesion datasets in a multi-task fashion and leverage their synergy by proposal fusion. Next, we propose strategies to mine missing annotations from partially-labeled datasets by exploiting clinical prior knowledge and cross-dataset knowledge transfer. Finally, we train our framework on four public lesion datasets and evaluate it on 800 manually-labeled sub-volumes in DeepLesion. Our method brings a relative improvement of 49% compared to the current state-of-the-art approach in the metric of average sensitivity. We have publicly released our manual 3D annotations of DeepLesion in https://github.com/viggin/DeepLesion_manual_test_set.