mlsanityData to 7 Oct 2026

Learning Spatio-Temporal Transformer for Visual Tracking

arXiv 2103.17154 ↗First posted 31 Mar 2021cs.CVcited by 348

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a new tracking architecture with an encoder-decoder transformer as the key component. The encoder models the global spatio-temporal feature dependencies between target objects and search regions, while the decoder learns a query embedding to predict the spatial positions of the target objects. Our method casts object tracking as a direct bounding box prediction problem, without using any proposals or predefined anchors. With the encoder-decoder transformer, the prediction of objects just uses a simple fully-convolutional network, which estimates the corners of objects directly. The whole method is end-to-end, does not need any postprocessing steps such as cosine window and bounding box smoothing, thus largely simplifying existing tracking pipelines. The proposed tracker achieves state-of-the-art performance on five challenging short-term and long-term benchmarks, while running at real-time speed, being 6x faster than Siam R-CNN. Code and models are open-sourced at https://github.com/researchmm/Stark.

Key facts. Cited by 348 papers in the map (data through 7 Oct 2026). Builds on 53 of its 69 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

53 of 69 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  10. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  11. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  12. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
38 more
  1. 2018SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networkscited by 445
  2. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  3. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 424
  4. 2021Transformer Trackingcited by 412
  5. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 408
  6. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  7. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  8. 2019Learning Discriminative Model Prediction for Trackingcited by 344
  9. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 342
  10. 2018ATOM: Accurate Tracking by Overlap Maximizationcited by 285
  11. 2018Fast Online Object Tracking and Segmentation: A Unifying Approachcited by 272
  12. 2016ECO: Efficient Convolution Operators for Trackingcited by 239
  13. 2020Probabilistic Regression for Visual Trackingcited by 231
  14. 2020Ocean: Object-aware Anchor-free Trackingcited by 212
  15. 2015Learning Multi-Domain Convolutional Neural Networks for Visual Trackingcited by 209
  16. 2020Siamese Box Adaptive Network for Visual Trackingcited by 189
  17. 2018Distractor-aware Siamese Networks for Visual Object Trackingcited by 187
  18. 2019SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelinescited by 177
  19. 2019Siam R-CNN: Visual Tracking by Re-Detectioncited by 174
  20. 2019Deeper and Wider Siamese Networks for Real-Time Visual Trackingcited by 156
  21. 2019SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Trackingcited by 147
  22. 2017Need for Speed: A Benchmark for Higher Frame Rate Object Trackingcited by 137
  23. 2020Deformable Siamese Attention Networks for Visual Object Trackingcited by 97
  24. 2017End-to-end representation learning for Correlation Filter based trackingcited by 90
  25. 2016Learning to Track at 100 FPS with Deep Regression Networkscited by 89
  26. 2020Revisiting the Sibling Head in Object Detectorcited by 82
  27. 2020Alpha-Refine: Boosting Tracking Performance by Precise Bounding Box Estimationcited by 81
  28. 2020High-Performance Long-Term Tracking with Meta-Updatercited by 79
  29. 2019Learning the Model Update for Siamese Trackerscited by 75
  30. 2019D3S -- A Discriminative Single Shot Segmentation Trackercited by 69
  31. 2019RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentationcited by 63
  32. 2020Tracking by Instance Detection: A Meta-Learning Approachcited by 61
  33. 2018Learning Dynamic Memory Networks for Object Trackingcited by 58
  34. 2019GlobalTrack: A Simple and Strong Baseline for Long-term Trackingcited by 57
  35. 2016Discriminative Correlation Filter with Channel and Spatial Reliabilitycited by 45
  36. 2018Unveiling the Power of Deep Trackingcited by 41
  37. 2019GradNet: Gradient-Guided Network for Visual Object Trackingcited by 41
  38. 2018Real-Time MDNetcited by 39

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 53; 2023: 79; 2024: 75; 2025: 71; 2026: 50.

348 papers.

Sort:

By area:

  1. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  2. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  3. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 278
  4. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  5. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  6. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  7. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  8. 2022DETRs with Hybrid Matchingcited by 149
  9. 2023Visual Prompt Multi-Modal Trackingcited by 141
  10. 2022AiATrack: Attention in Attention for Transformer Visual Trackingcited by 137
  11. 2024ODTrack: Online Dense Temporal Token Learning for Visual Trackingcited by 129
  12. 2021VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flowscited by 128
38 more
  1. 2021SwinTrack: A Simple and Strong Baseline for Transformer Trackingcited by 126
  2. 2022Backbone is All Your Need: A Simplified Architecture for Visual Object Trackingcited by 119
  3. 2022Transforming Model Prediction for Trackingcited by 117
  4. 2021SeqFormer: Sequential Transformer for Video Instance Segmentationcited by 98
  5. 2022Towards Grand Unification of Object Trackingcited by 93
  6. 2023Generalized Relation Modeling for Transformer Trackingcited by 93
  7. 2024SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memorycited by 91
  8. 2022Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baselinecited by 90
  9. 2023Robust Object Modeling for Visual Trackingcited by 80
  10. 2024Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformerscited by 76
  11. 2022Transformer Tracking with Cyclic Shifting Window Attentioncited by 75
  12. 2023Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baselinecited by 73
  13. 2023HIPTrack: Visual Tracking with Historical Promptscited by 71
  14. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  15. 2022TCTrack: Temporal Contexts for Aerial Trackingcited by 69
  16. 2023Single-Model and Any-Modality for Video Object Trackingcited by 64
  17. 2024OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuningcited by 64
  18. 2022Video Transformers: A Surveycited by 63
  19. 2024A Distractor-Aware Memory for Visual Object Tracking with SAM2cited by 63
  20. 2022Correlation-Aware Deep Trackingcited by 62
  21. 2023Towards Unified Token Learning for Vision-Language Trackingcited by 62
  22. 2023Joint Visual Grounding and Tracking with Natural Language Specificationcited by 59
  23. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 59
  24. 2022Prompting for Multi-Modal Trackingcited by 58
  25. 2021Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Predictioncited by 57
  26. 2023MixFormerV2: Efficient Fully Transformer Trackingcited by 56
  27. 2022APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Trackingcited by 55
  28. 2024Exploiting Multimodal Spatial-temporal Patterns for Video Object Trackingcited by 55
  29. 2023Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Trackingcited by 53
  30. 2023CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentationcited by 52
  31. 2022Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metriccited by 51
  32. 2023Compact Transformer Tracker with Correlative Masked Modelingcited by 50
  33. 2024Explicit Visual Prompts for Visual Object Trackingcited by 50
  34. 2021Efficient Visual Tracking with Exemplar Transformerscited by 48
  35. 2022Divert More Attention to Vision-Language Trackingcited by 47
  36. 2023Bi-directional Adapter for Multi-modal Trackingcited by 46
  37. 2021Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlookcited by 45
  38. 2023Annotation-free Audio-Visual Segmentationcited by 44

50 most cited of 348 All 348 (JSON)