# PTSEFormer: Progressive Temporal-Spatial Enhanced TransFormer Towards Video Object Detection

arXiv: https://arxiv.org/abs/2209.02242 · HTML: https://mlsanity.com/paper/2209.02242/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 6 Sep 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [Flow-Guided Feature Aggregation for Video Object Detection](https://mlsanity.com/paper/1703.10025/) (2017)
- [End-to-end people detection in crowded scenes](https://mlsanity.com/paper/1506.04878/) (2015)
- [Detect to Track and Track to Detect](https://mlsanity.com/paper/1710.03958/) (2017)
- [Memory Enhanced Global-Local Aggregation for Video Object Detection](https://mlsanity.com/paper/2003.12063/) (2020)
- [Sequence Level Semantics Aggregation for Video Object Detection](https://mlsanity.com/paper/1907.06390/) (2019)
- [TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers](https://mlsanity.com/paper/2201.05047/) (2022)
- [Relation Distillation Networks for Video Object Detection](https://mlsanity.com/paper/1908.09511/) (2019)
- [HiFT: Hierarchical Feature Transformer for Aerial Tracking](https://mlsanity.com/paper/2108.00202/) (2021)
- [Seq-NMS for Video Object Detection](https://mlsanity.com/paper/1602.08465/) (2016)
- [T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos](https://mlsanity.com/paper/1604.02532/) (2016)
- [Temporal RoI Align for Video Object Recognition](https://mlsanity.com/paper/2109.03495/) (2021)
- [Progressive Sparse Local Attention for Video object detection](https://mlsanity.com/paper/1903.09126/) (2019)
- [Optimizing Video Object Detection via a Scale-Time Lattice](https://mlsanity.com/paper/1804.05472/) (2018)

## Built on by (most cited first, 17 of 17)

- [TransFlow: Transformer as Flow Learner](https://mlsanity.com/paper/2304.11523/) (2023)
- [Transformers in Small Object Detection: A Benchmark and Survey of State-of-the-Art](https://mlsanity.com/paper/2309.04902/) (2023)
- [LongShortNet: Exploring Temporal and Semantic Features Fusion in Streaming Perception](https://mlsanity.com/paper/2210.15518/) (2022)
- [FAQ: Feature Aggregated Queries for Transformer-based Video Object Detectors](https://mlsanity.com/paper/2303.08319/) (2023)
- [Objects do not disappear: Video object detection by single-frame object location anticipation](https://mlsanity.com/paper/2308.04770/) (2023)
- [Video Instance Segmentation in an Open-World](https://mlsanity.com/paper/2304.01200/) (2023)
- [Practical Video Object Detection via Feature Selection and Aggregation](https://mlsanity.com/paper/2407.19650/) (2024)
- [Source-free Domain Adaptation for Video Object Detection Under Adverse Image Conditions](https://mlsanity.com/paper/2404.15252/) (2024)
- [Mining Negative Temporal Contexts For False Positive Suppression In Real-Time Ultrasound Lesion Detection](https://mlsanity.com/paper/2305.18060/) (2023)
- [SSGA-Net: Stepwise Spatial Global-local Aggregation Networks for for Autonomous Driving](https://mlsanity.com/paper/2405.18857/) (2024)
- [Map-Free Trajectory Prediction with Map Distillation and Hierarchical Encoding](https://mlsanity.com/paper/2411.10961/) (2024)
- [Beyond Boxes: Mask-Guided Spatio-Temporal Feature Aggregation for Video Object Detection](https://mlsanity.com/paper/2412.04915/) (2024)
- [Improving Token-based Object Detection with Video](https://mlsanity.com/paper/2506.22562/) (2025)
- [Sketch-based Video Object Localization](https://mlsanity.com/paper/2304.00450/) (2023)
- [GloTSFormer: Global Video Text Spotting Transformer](https://mlsanity.com/paper/2401.03694/) (2024)
- [Multimodal Spatio-temporal Graph Learning for Alignment-free RGBT Video Object Detection](https://mlsanity.com/paper/2504.11779/) (2025)
- [Contrastive Learning through Auxiliary Branch for Video Object Detection](https://mlsanity.com/paper/2508.20551/) (2025)

## Abstract (arXiv)

Recent years have witnessed a trend of applying context frames to boost the performance of object detection as video object detection. Existing methods usually aggregate features at one stroke to enhance the feature. These methods, however, usually lack spatial information from neighboring frames and suffer from insufficient feature aggregation. To address the issues, we perform a progressive way to introduce both temporal information and spatial information for an integrated enhancement. The temporal information is introduced by the temporal feature aggregation model (TFAM), by conducting an attention mechanism between the context frames and the target frame (i.e., the frame to be detected). Meanwhile, we employ a Spatial Transition Awareness Model (STAM) to convey the location transition information between each context frame and target frame. Built upon a transformer-based detector DETR, our PTSEFormer also follows an end-to-end fashion to avoid heavy post-processing procedures while achieving 88.1% mAP on the ImageNet VID dataset. Codes are available at https://github.com/Hon-Wong/PTSEFormer.