# Prediction of Rectal Cancer Regrowth from Longitudinal Endoscopy

arXiv: https://arxiv.org/abs/2605.12855 · HTML: https://mlsanity.com/paper/2605.12855/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 55 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train](https://mlsanity.com/paper/2306.16741/) (2023)
- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reports](https://mlsanity.com/paper/2306.08749/) (2023)
- [A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints](https://mlsanity.com/paper/2304.09941/) (2023)
- [COTR: Convolution in Transformer Network for End to End Polyp Detection](https://mlsanity.com/paper/2105.10925/) (2021)
- [GAN Inversion for Data Augmentation to Improve Colonoscopy Lesion Classification](https://mlsanity.com/paper/2205.02840/) (2022)
- [Progressively refined deep joint registration segmentation (ProRSeg) of gastrointestinal organs at risk: Application to MRI and cone-beam CT](https://mlsanity.com/paper/2210.14297/) (2022)
- [Swin transformers are robust to distribution and concept drift in endoscopy-based longitudinal rectal cancer assessment](https://mlsanity.com/paper/2405.03762/) (2024)

## Abstract (arXiv)

Clinical trial studies indicate benefit of watch-and-wait (WW) surveillance for patients with rectal cancer showing a complete or near clinical response (CR) directly after treatment (restaging). However, there are no objectively accurate methods to early detect local tumor regrowth (LR) in patients undergoing WW from follow-up exams. Hence, we developed Temporal Rectal Endoscopy Cross-attention (TREX), a longitudinal deep learning approach that combines pairs of images acquired at restaging and follow-up to distinguish CR from LR. TREX uses pretrained Swin Transformers in a siamese setting to extract features from longitudinal images and dual cross-attention to combine the features without spatial co-registration between image pairs. TREX and Swin-based baselines were trained under two settings: (a) detecting LR or CR at the last available follow-up and (b) early detection of LR at 3--6, 6--12, and 12--24 months before clinical confirmation. TREX achieved the highest accuracy in detecting LR with a high sensitivity of 97% ± 6% and a balanced accuracy of 90% ± 3%, and outperformed all baselines in early detection at both 3--6 (74% ± 1%) and 6--12 months (62% ± 4%) prior to clinical detection. Clinical validation via a surgeon survey showed that TREX matched attending-level overall accuracy (TREX: 86.21% vs. Clinicians: 87.84% ± 1.28%). Finally, we explored TREX's ability to predict treatment response by combining pre-treatment (pre-TNT) and restaging endoscopies, achieving a balanced accuracy of 73% ± 12%. These results show that longitudinal deep learning analysis of endoscopy may improve surveillance and enable earlier identification of rectal cancer regrowth.