# Spatiotemporal Transformer for Video-based Person Re-identification

arXiv: https://arxiv.org/abs/2103.16469 · HTML: https://mlsanity.com/paper/2103.16469/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)](https://mlsanity.com/paper/1711.09349/) (2017)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Omni-Scale Feature Learning for Person Re-Identification](https://mlsanity.com/paper/1905.00953/) (2019)
- [Learning Discriminative Features with Multiple Granularities for Person Re-Identification](https://mlsanity.com/paper/1804.01438/) (2018)
- [Pose-driven Deep Convolutional Model for Person Re-identification](https://mlsanity.com/paper/1709.08325/) (2017)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [A Pose-Sensitive Embedding for Person Re-Identification with Expanded Cross Neighborhood Re-Ranking](https://mlsanity.com/paper/1711.10378/) (2017)
- [Learning Generalisable Omni-Scale Representations for Person Re-Identification](https://mlsanity.com/paper/1910.06827/) (2019)
- [LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention](https://mlsanity.com/paper/2004.01389/) (2020)
- [Appearance-Preserving 3D Convolution for Video-based Person Re-identification](https://mlsanity.com/paper/2007.08434/) (2020)
- [Diversity Regularized Spatiotemporal Attention for Video-based Person Re-identification](https://mlsanity.com/paper/1803.09882/) (2018)
- [STA: Spatial-Temporal Attention for Large-Scale Video-based Person Re-Identification](https://mlsanity.com/paper/1811.04129/) (2018)
- [Watching You: Global-guided Reciprocal Learning for Video-based Person Re-identification](https://mlsanity.com/paper/2103.04337/) (2021)
- [Multi-Granularity Reference-Aided Attentive Feature Aggregation for Video-based Person Re-identification](https://mlsanity.com/paper/2003.12224/) (2020)
- [Joint Visual and Temporal Consistency for Unsupervised Domain Adaptive Person Re-Identification](https://mlsanity.com/paper/2007.10854/) (2020)
- [Multi-scale 3D Convolution Network for Video Based Person Re-Identification](https://mlsanity.com/paper/1811.07468/) (2018)
- [Second-order Non-local Attention Networks for Person Re-identification](https://mlsanity.com/paper/1909.00295/) (2019)

## Built on by (most cited first, 15 of 15)

- [TF-CLIP: Learning Text-free CLIP for Video-based Person Re-Identification](https://mlsanity.com/paper/2312.09627/) (2023)
- [HAT: Hierarchical Aggregation Transformers for Person Re-identification](https://mlsanity.com/paper/2107.05946/) (2021)
- [Self-Supervised Pre-Training for Transformer-Based Person Re-Identification](https://mlsanity.com/paper/2111.12084/) (2021)
- [Transformer for Object Re-Identification: A Survey](https://mlsanity.com/paper/2401.06960/) (2024)
- [Recent Advances in Vision Transformer: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2203.01536/) (2022)
- [Deeply-Coupled Convolution-Transformer with Spatial-temporal Complementary Learning for Video-based Person Re-identification](https://mlsanity.com/paper/2304.14122/) (2023)
- [Video-based Person Re-identification with Long Short-Term Representation Learning](https://mlsanity.com/paper/2308.03703/) (2023)
- [Multi-Stage Spatio-Temporal Aggregation Transformer for Video Person Re-identification](https://mlsanity.com/paper/2301.00531/) (2023)
- [Deep Learning for Video-based Person Re-Identification: A Survey](https://mlsanity.com/paper/2303.11332/) (2023)
- [Adversarial Self-Attack Defense and Spatial-Temporal Relation Mining for Visible-Infrared Video Person Re-Identification](https://mlsanity.com/paper/2307.03903/) (2023)
- [Seeing Across Time and Views: Multi-Temporal Cross-View Learning for Robust Video Person Re-Identification](https://mlsanity.com/paper/2511.02564/) (2025)
- [TVPR: Text-to-Video Person Retrieval and a New Benchmark](https://mlsanity.com/paper/2307.07184/) (2023)
- [Object Re-identification via Spatial-temporal Fusion Networks and Causal Identity Matching](https://mlsanity.com/paper/2408.05558/) (2024)
- [KeyRe-ID: Keypoint-Guided Person Re-Identification using Part-Aware Representation in Videos](https://mlsanity.com/paper/2507.07393/) (2025)
- [Generating Humanless Environment Walkthroughs from Egocentric Walking Tour Videos](https://mlsanity.com/paper/2603.29036/) (2026)

## Abstract (arXiv)

Recently, the Transformer module has been transplanted from natural language processing to computer vision. This paper applies the Transformer to video-based person re-identification, where the key issue is to extract the discriminative information from a tracklet. We show that, despite the strong learning ability, the vanilla Transformer suffers from an increased risk of over-fitting, arguably due to a large number of attention parameters and insufficient training data. To solve this problem, we propose a novel pipeline where the model is pre-trained on a set of synthesized video data and then transferred to the downstream domains with the perception-constrained Spatiotemporal Transformer (STT) module and Global Transformer (GT) module. The derived algorithm achieves significant accuracy gain on three popular video-based person re-identification benchmarks, MARS, DukeMTMC-VideoReID, and LS-VID, especially when the training and testing data are from different domains. More importantly, our research sheds light on the application of the Transformer on highly-structured visual data.