# Multi-frame Collaboration for Effective Endoscopic Video Polyp Detection via Spatial-Temporal Feature Transformation

arXiv: https://arxiv.org/abs/2107.03609 · HTML: https://mlsanity.com/paper/2107.03609/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 8 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [Flow-Guided Feature Aggregation for Video Object Detection](https://mlsanity.com/paper/1703.10025/) (2017)
- [Memory Enhanced Global-Local Aggregation for Video Object Detection](https://mlsanity.com/paper/2003.12063/) (2020)
- [Region Proposal by Guided Anchoring](https://mlsanity.com/paper/1901.03278/) (2019)
- [Relation Distillation Networks for Video Object Detection](https://mlsanity.com/paper/1908.09511/) (2019)
- [Object Detection in Video with Spatiotemporal Sampling Networks](https://mlsanity.com/paper/1803.05549/) (2018)
- [SenseCare: A Research Platform for Medical Image Informatics and Interactive 3D Visualization](https://mlsanity.com/paper/2004.07031/) (2020)

## Built on by (most cited first, 15 of 15)

- [Video Polyp Segmentation: A Deep Learning Perspective](https://mlsanity.com/paper/2203.14291/) (2022)
- [Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train](https://mlsanity.com/paper/2306.16741/) (2023)
- [HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition](https://mlsanity.com/paper/2405.10075/) (2024)
- [ArSDM: Colonoscopy Images Synthesis with Adaptive Refinement Semantic Diffusion Models](https://mlsanity.com/paper/2309.01111/) (2023)
- [Frontiers in Intelligent Colonoscopy](https://mlsanity.com/paper/2410.17241/) (2024)
- [A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trends](https://mlsanity.com/paper/2311.18373/) (2023)
- [YONA: You Only Need One Adjacent Reference-frame for Accurate and Fast Video Polyp Detection](https://mlsanity.com/paper/2306.03686/) (2023)
- [ECC-PolypDet: Enhanced CenterNet with Contrastive Learning for Automatic Polyp Detection](https://mlsanity.com/paper/2401.04961/) (2024)
- [MAST: Video Polyp Segmentation with a Mixture-Attention Siamese Transformer](https://mlsanity.com/paper/2401.12439/) (2024)
- [In Defense of Kalman Filtering for Polyp Tracking from Colonoscopy Videos](https://mlsanity.com/paper/2201.11450/) (2022)
- [Let Video Teaches You More: Video-to-Image Knowledge Distillation using DEtection TRansformer for Medical Video Lesion Detection](https://mlsanity.com/paper/2408.14051/) (2024)
- [Universal Medical Image Representation Learning with Compositional Decoders](https://mlsanity.com/paper/2409.19890/) (2024)
- [AVPDN: Learning Motion-Robust and Scale-Adaptive Representations for Video-Based Polyp Detection](https://mlsanity.com/paper/2508.03458/) (2025)
- [A multi-center analysis of deep learning methods for video polyp detection and segmentation](https://mlsanity.com/paper/2603.04288/) (2026)
- [Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis](https://mlsanity.com/paper/2603.25778/) (2026)

## Abstract (arXiv)

Precise localization of polyp is crucial for early cancer screening in gastrointestinal endoscopy. Videos given by endoscopy bring both richer contextual information as well as more challenges than still images. The camera-moving situation, instead of the common camera-fixed-object-moving one, leads to significant background variation between frames. Severe internal artifacts (e.g. water flow in the human body, specular reflection by tissues) can make the quality of adjacent frames vary considerately. These factors hinder a video-based model to effectively aggregate features from neighborhood frames and give better predictions. In this paper, we present Spatial-Temporal Feature Transformation (STFT), a multi-frame collaborative framework to address these issues. Spatially, STFT mitigates inter-frame variations in the camera-moving situation with feature alignment by proposal-guided deformable convolutions. Temporally, STFT proposes a channel-aware attention module to simultaneously estimate the quality and correlation of adjacent frames for adaptive feature aggregation. Empirical studies and superior results demonstrate the effectiveness and stability of our method. For example, STFT improves the still image baseline FCOS by 10.6% and 20.6% on the comprehensive F1-score of the polyp localization task in CVC-Clinic and ASUMayo datasets, respectively, and outperforms the state-of-the-art video-based method by 3.6% and 8.0%, respectively. Code is available at urlhttps://github.com/lingyunwu14/STFT.