mlsanityData to 7 Oct 2026

TrackFormer: Multi-Object Tracking with Transformers

arXiv 2101.02702 ↗First posted 7 Jan 2021cs.CVcited by 515

Not on the map. Why?

Abstract (arXiv)

The challenging task of multi-object tracking (MOT) requires simultaneous reasoning about track initialization, identity, and spatio-temporal trajectories. We formulate this task as a frame-to-frame set prediction problem and introduce TrackFormer, an end-to-end trainable MOT approach based on an encoder-decoder Transformer architecture. Our model achieves data association between frames via attention by evolving a set of track predictions through a video sequence. The Transformer decoder initializes new tracks from static object queries and autoregressively follows existing tracks in space and time with the conceptually new and identity preserving track queries. Both query types benefit from self- and encoder-decoder attention on global frame-level features, thereby omitting any additional graph optimization or modeling of motion and/or appearance. TrackFormer introduces a new tracking-by-attention paradigm and while simple in its design is able to achieve state-of-the-art performance on the task of multi-object tracking (MOT17 and MOT20) and segmentation (MOTS20). The code is available at https://github.com/timmeinhardt/trackformer .

Key facts. Cited by 515 papers in the map (data through 7 Oct 2026). Builds on 36 of its 73 references that are in the map. First posted on arXiv 7 Jan 2021.

Builds on

36 of 73 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  9. 2018Image Transformercited by 770
  10. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 685
  11. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  12. 2020FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Trackingcited by 489
24 more
  1. 2020Tracking Objects as Pointscited by 483
  2. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  3. 2019Tracking without bells and whistlescited by 382
  4. 2020MOT20: A benchmark for multi object tracking in crowded scenescited by 355
  5. 2018CrowdHuman: A Benchmark for Detecting Human in a Crowdcited by 283
  6. 2019MOTS: Multi-Object Tracking and Segmentationcited by 222
  7. 2020Quasi-Dense Similarity Learning for Multiple Object Trackingcited by 220
  8. 2019Learning a Neural Solver for Multiple Object Trackingcited by 198
  9. 2015End-to-end people detection in crowded scenescited by 140
  10. 2020MOTChallenge: A Benchmark for Single-Camera Multiple Target Trackingcited by 134
  11. 2017Detect to Track and Track to Detectcited by 131
  12. 2020Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Trackingcited by 115
  13. 2018Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identificationcited by 96
  14. 2018Features for Multi-Target Multi-Camera Tracking and Re-Identificationcited by 88
  15. 2018Learning to Detect and Track Visible and Occluded Body Joints in a Virtual Worldcited by 81
  16. 2016Learning by tracking: Siamese CNN for robust target associationcited by 79
  17. 2019Online Multi-Object Tracking with Dual Matching Attention Networkscited by 77
  18. 2020TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Modelcited by 70
  19. 2019FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Trackingcited by 65
  20. 2020Lifted Disjoint Paths with Application in Multiple Object Trackingcited by 55
  21. 2017Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanismcited by 48
  22. 2020Segment as Points for Efficient Online Multi-Object Tracking and Segmentationcited by 36
  23. 2019Learning Multi-Object Tracking and Segmentation from Automatic Annotationscited by 27
  24. 2017Track, then Decide: Category-Agnostic Vision-based Multi-Object Trackingcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 101; 2023: 113; 2024: 120; 2025: 86; 2026: 54.

515 papers.

Sort:

By area:

  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  2. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  3. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  4. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  5. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  6. 2022Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Trackingcited by 345
  7. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  8. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  9. 2021Conditional Object-Centric Learning from Videocited by 247
  10. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  11. 2021DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motioncited by 213
  12. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
38 more
  1. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  2. 2022BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimationcited by 166
  3. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  4. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 165
  5. 2021Instances as Queriescited by 157
  6. 2021Track to Detect and Segment: An Online Multi-Object Trackercited by 156
  7. 2022DETRs with Hybrid Matchingcited by 149
  8. 2022MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectorscited by 148
  9. 2022Rethinking Semantic Segmentation: A Prototype Viewcited by 140
  10. 2022AiATrack: Attention in Attention for Transformer Visual Trackingcited by 137
  11. 2023Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identificationcited by 130
  12. 2023SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenescited by 130
  13. 2024ODTrack: Online Dense Temporal Token Learning for Visual Trackingcited by 129
  14. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  15. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  16. 2022MeMOT: Multi-Object Tracking with Memorycited by 113
  17. 2022Global Tracking Transformerscited by 110
  18. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  19. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 107
  20. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  21. 2021SeqFormer: Sequential Transformer for Video Instance Segmentationcited by 98
  22. 2022In Defense of Online Models for Video Instance Segmentationcited by 97
  23. 2022MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Trainingcited by 94
  24. 2021TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Trackingcited by 93
  25. 2022Towards Grand Unification of Object Trackingcited by 93
  26. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  27. 2023Referring Multi-Object Trackingcited by 91
  28. 2022MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queriescited by 90
  29. 2023TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environmentscited by 89
  30. 2023MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Trackingcited by 86
  31. 2022Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentationcited by 79
  32. 2022TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformerscited by 78
  33. 2023DVIS: Decoupled Video Instance Segmentation Frameworkcited by 77
  34. 2024Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformerscited by 76
  35. 2021HiFT: Hierarchical Feature Transformer for Aerial Trackingcited by 72
  36. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  37. 2023Sparse4D v3: Advancing End-to-End 3D Detection and Trackingcited by 69
  38. 2023Cross Modal Transformer: Towards Fast and Robust 3D Object Detectioncited by 68

50 most cited of 515 All 515 (JSON)