# Deep Lesion Tracker: Monitoring Lesions in 4D Longitudinal Imaging Studies

arXiv: https://arxiv.org/abs/2012.04872 · HTML: https://mlsanity.com/paper/2012.04872/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 25 of 25)

- [The ULS23 Challenge: a Baseline Model and Benchmark Dataset for 3D Universal Lesion Segmentation in Computed Tomography](https://mlsanity.com/paper/2406.05231/) (2024)
- [3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks](https://mlsanity.com/paper/2506.11147/) (2025)
- [SATr: Slice Attention with Transformer for Universal Lesion Detection](https://mlsanity.com/paper/2203.07373/) (2022)
- [Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss](https://mlsanity.com/paper/2105.01218/) (2021)
- [SAME++: A Self-supervised Anatomical eMbeddings Enhanced medical image registration framework using stable sampling and regularized transformation](https://mlsanity.com/paper/2311.14986/) (2023)
- [Lesion Segmentation and RECIST Diameter Prediction via Click-driven Attention and Dual-path Connection](https://mlsanity.com/paper/2105.01828/) (2021)
- [Transformer Lesion Tracker](https://mlsanity.com/paper/2206.06252/) (2022)
- [SAM++: Enhancing Anatomic Matching using Semantic Information and Structural Inference](https://mlsanity.com/paper/2306.13988/) (2023)
- [Accurate and Robust Lesion RECIST Diameter Prediction and Segmentation with Transformers](https://mlsanity.com/paper/2208.13113/) (2022)
- [A Hierarchical Descriptor Framework for On-the-Fly Anatomical Location Matching between Longitudinal Studies](https://mlsanity.com/paper/2308.07337/) (2023)
- [landmarker: a Toolkit for Anatomical Landmark Localization in 2D/3D Images](https://mlsanity.com/paper/2501.10098/) (2025)
- [Improving Small Lesion Segmentation in CT Scans using Intensity Distribution Supervision: Application to Small Bowel Carcinoid Tumor](https://mlsanity.com/paper/2207.14700/) (2022)
- [Improving Segmentation and Detection of Lesions in CT Scans Using Intensity Distribution Supervision](https://mlsanity.com/paper/2307.05804/) (2023)
- [AI-Enhanced Virtual Reality in Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2402.03093/) (2024)
- [ST-NeRP: Spatial-Temporal Neural Representation Learning with Prior Embedding for Patient-specific Imaging Study](https://mlsanity.com/paper/2410.19283/) (2024)
- [Label Filling via Mixed Supervision for Medical Image Segmentation from Noisy Annotations](https://mlsanity.com/paper/2410.16057/) (2024)
- [Class Imbalance Correction for Improved Universal Lesion Detection and Tagging in CT](https://mlsanity.com/paper/2504.05591/) (2025)
- [Towards AI Lesion Tracking in PET/CT Imaging: A Siamese-based CNN Pipeline applied on PSMA PET/CT Scans](https://mlsanity.com/paper/2406.09327/) (2024)
- [LinGuinE: Longitudinal Guidance Estimation for Volumetric Tumour Segmentation](https://mlsanity.com/paper/2506.06092/) (2025)
- [MammoTracker: Mask-Guided Lesion Tracking in Temporal Mammograms](https://mlsanity.com/paper/2507.00328/) (2025)
- [Consistent Point Matching](https://mlsanity.com/paper/2507.23609/) (2025)
- [CT Scans As Video: Efficient Intracranial Hemorrhage Detection Using Multi-Object Tracking](https://mlsanity.com/paper/2601.02521/) (2026)
- [Schr"odinger-Inspired Time-Evolution for 4D Deformation Forecasting](https://mlsanity.com/paper/2602.00661/) (2026)
- [GLIDE-Reg: Global-to-Local Deformable Registration Using Co-Optimized Foundation and Handcrafted Features](https://mlsanity.com/paper/2603.00218/) (2026)
- [Reliability-Centered Evaluation of Sparse Longitudinal CT Lesion-Size Forecasting with Conformal Interval Calibration and Gompertz-Inspired Regularization](https://mlsanity.com/paper/2609.21197/) (2026)

## Abstract (arXiv)

Monitoring treatment response in longitudinal studies plays an important role in clinical practice. Accurately identifying lesions across serial imaging follow-up is the core to the monitoring procedure. Typically this incorporates both image and anatomical considerations. However, matching lesions manually is labor-intensive and time-consuming. In this work, we present deep lesion tracker (DLT), a deep learning approach that uses both appearance- and anatomical-based signals. To incorporate anatomical constraints, we propose an anatomical signal encoder, which prevents lesions being matched with visually similar but spurious regions. In addition, we present a new formulation for Siamese networks that avoids the heavy computational loads of 3D cross-correlation. To present our network with greater varieties of images, we also propose a self-supervised learning (SSL) strategy to train trackers with unpaired images, overcoming barriers to data collection. To train and evaluate our tracker, we introduce and release the first lesion tracking benchmark, consisting of 3891 lesion pairs from the public DeepLesion database. The proposed method, DLT, locates lesion centers with a mean error distance of 7 mm. This is 5% better than a leading registration algorithm while running 14 times faster on whole CT volumes. We demonstrate even greater improvements over detector or similarity-learning alternatives. DLT also generalizes well on an external clinical test set of 100 longitudinal studies, achieving 88% accuracy. Finally, we plug DLT into an automatic tumor monitoring workflow where it leads to an accuracy of 85% in assessing lesion treatment responses, which is only 0.46% lower than the accuracy of manual inputs.