# Transformer Lesion Tracker

arXiv: https://arxiv.org/abs/2206.06252 · HTML: https://mlsanity.com/paper/2206.06252/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [An Unsupervised Learning Model for Deformable Medical Image Registration](https://mlsanity.com/paper/1802.02604/) (2018)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Deep Lesion Graphs in the Wild: Relationship Learning and Organization of Significant Radiology Image Findings in a Diverse Large-scale Lesion Database](https://mlsanity.com/paper/1711.10535/) (2017)
- [Learning from Multiple Datasets with Heterogeneous and Partial Labels for Universal Lesion Detection in CT](https://mlsanity.com/paper/2009.02577/) (2020)
- [Deep Lesion Tracker: Monitoring Lesions in 4D Longitudinal Imaging Studies](https://mlsanity.com/paper/2012.04872/) (2020)
- [Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images: Learning from Radiology Reports and Label Ontology](https://mlsanity.com/paper/1904.04661/) (2019)
- [Co-Heterogeneous and Adaptive Segmentation from Multi-Source and Multi-Phase CT Imaging Data: A Study on Pathological Liver and Lesion Segmentation](https://mlsanity.com/paper/2005.13201/) (2020)
- [Deep Volumetric Universal Lesion Detection using Light-Weight Pseudo 3D Convolution and Surface Point Regression](https://mlsanity.com/paper/2008.13254/) (2020)
- [Re-Identification and Growth Detection of Pulmonary Nodules without Image Registration Using 3D Siamese Neural Networks](https://mlsanity.com/paper/1912.10525/) (2019)
- [Siamese Networks with Location Prior for Landmark Tracking in Liver Ultrasound Sequences](https://mlsanity.com/paper/1901.08109/) (2019)
- [Attentive CT Lesion Detection Using Deep Pyramid Inference with Multi-Scale Booster](https://mlsanity.com/paper/1907.03958/) (2019)

## Built on by (most cited first, 6 of 6)

- [SAM++: Enhancing Anatomic Matching using Semantic Information and Structural Inference](https://mlsanity.com/paper/2306.13988/) (2023)
- [Class Imbalance Correction for Improved Universal Lesion Detection and Tagging in CT](https://mlsanity.com/paper/2504.05591/) (2025)
- [Towards AI Lesion Tracking in PET/CT Imaging: A Siamese-based CNN Pipeline applied on PSMA PET/CT Scans](https://mlsanity.com/paper/2406.09327/) (2024)
- [Unbalanced optimal transport for robust longitudinal lesion evolution with registration-aware and appearance-guided priors](https://mlsanity.com/paper/2602.09933/) (2026)
- [Longitudinal tracking of multiple sclerosis lesions in the spinal cord: A validation study](https://mlsanity.com/paper/2609.09424/) (2026)
- [NoduleFlow: Automatic 3D Lung Nodule Detection and Segmentation](https://mlsanity.com/paper/2610.16548/) (2026)

## Abstract (arXiv)

Evaluating lesion progression and treatment response via longitudinal lesion tracking plays a critical role in clinical practice. Automated approaches for this task are motivated by prohibitive labor costs and time consumption when lesion matching is done manually. Previous methods typically lack the integration of local and global information. In this work, we propose a transformer-based approach, termed Transformer Lesion Tracker (TLT). Specifically, we design a Cross Attention-based Transformer (CAT) to capture and combine both global and local information to enhance feature extraction. We also develop a Registration-based Anatomical Attention Module (RAAM) to introduce anatomical information to CAT so that it can focus on useful feature knowledge. A Sparse Selection Strategy (SSS) is presented for selecting features and reducing memory footprint in Transformer training. In addition, we use a global regression to further improve model performance. We conduct experiments on a public dataset to show the superiority of our method and find that our model performance has improved the average Euclidean center error by at least 14.3% (6mm vs. 7mm) compared with the state-of-the-art (SOTA). Code is available at https://github.com/TangWen920812/TLT.