mlsanityData to 7 Oct 2026

MOTR: End-to-End Multiple-Object Tracking with Transformer

arXiv 2105.03247 ↗First posted 7 May 2021cs.CVcited by 379

Not on the map. Why?

Abstract (arXiv)

Temporal modeling of objects is a key challenge in multiple object tracking (MOT). Existing methods track by associating detections through motion-based and appearance-based similarity heuristics. The post-processing nature of association prevents end-to-end exploitation of temporal variations in video sequence. In this paper, we propose MOTR, which extends DETR and introduces track query to model the tracked instances in the entire video. Track query is transferred and updated frame-by-frame to perform iterative prediction over time. We propose tracklet-aware label assignment to train track queries and newborn object queries. We further propose temporal aggregation network and collective average loss to enhance temporal relation modeling. Experimental results on DanceTrack show that MOTR significantly outperforms state-of-the-art method, ByteTrack by 6.5% on HOTA metric. On MOT17, MOTR outperforms our concurrent works, TrackFormer and TransTrack, on association performance. MOTR can serve as a stronger baseline for future research on temporal modeling and Transformer-based trackers. Code is available at https://github.com/megvii-research/MOTR.

Key facts. Cited by 379 papers in the map (data through 7 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

45 of 62 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2018YOLOv3: An Incremental Improvementcited by 2,652
  11. 2017Non-local Neural Networkscited by 2,290
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
33 more
  1. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  2. 2019Objects as Pointscited by 1,172
  3. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 908
  4. 2020Pre-Trained Image Processing Transformercited by 891
  5. 2016Simple Online and Realtime Trackingcited by 862
  6. 2017Focal Loss for Dense Object Detectioncited by 749
  7. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  8. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  9. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  10. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  11. 2020FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Trackingcited by 489
  12. 2020Tracking Objects as Pointscited by 483
  13. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  14. 2019Tracking without bells and whistlescited by 382
  15. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  16. 2019Towards Real-Time Multi-Object Trackingcited by 311
  17. 2018CrowdHuman: A Benchmark for Detecting Human in a Crowdcited by 283
  18. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  19. 2020Quasi-Dense Similarity Learning for Multiple Object Trackingcited by 220
  20. 2021Track to Detect and Segment: An Online Multi-Object Trackercited by 156
  21. 2020Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Trackingcited by 115
  22. 2018Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identificationcited by 96
  23. 2020Feature Pyramid Transformercited by 94
  24. 2021TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Trackingcited by 93
  25. 2018Deep Affinity Network for Multiple Object Trackingcited by 82
  26. 2016Learning by tracking: Siamese CNN for robust target associationcited by 79
  27. 2021Multiple Object Tracking with Correlation Learningcited by 67
  28. 2017Deep Network Flow for Multi-Object Trackingcited by 45
  29. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
  30. 2018Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Trackingcited by 40
  31. 2020Multi-Object Tracking with Siamese Track-RCNNcited by 14
  32. 2018Trajectory Factory: Tracklet Cleaving and Re-connection by Deep Siamese Bi-GRU for Multiple Object Trackingcited by 7
  33. 2020Learning to associate detections for real-time multiple object trackingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 66; 2023: 79; 2024: 105; 2025: 80; 2026: 45.

379 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  3. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 526
  4. 2024YOLOv10: Real-Time End-to-End Object Detectioncited by 508
  5. 2022Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Trackingcited by 345
  6. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  7. 2022BoT-SORT: Robust Associations Multi-Pedestrian Trackingcited by 313
  8. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  9. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  10. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  11. 2021DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motioncited by 213
  12. 2022StrongSORT: Make DeepSORT Great Againcited by 213
38 more
  1. 2022DETRs with Hybrid Matchingcited by 149
  2. 2022MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectorscited by 148
  3. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  4. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  5. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  6. 2023Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identificationcited by 130
  7. 2023SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenescited by 130
  8. 2024ODTrack: Online Dense Temporal Token Learning for Visual Trackingcited by 129
  9. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  10. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  11. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  12. 2022MeMOT: Multi-Object Tracking with Memorycited by 113
  13. 2022Global Tracking Transformerscited by 110
  14. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
  15. 2022MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Trainingcited by 94
  16. 2022Towards Grand Unification of Object Trackingcited by 93
  17. 2023Referring Multi-Object Trackingcited by 91
  18. 2022MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queriescited by 90
  19. 2023MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Trackingcited by 86
  20. 2022Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentationcited by 79
  21. 2024Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformerscited by 76
  22. 2023Sparse4D v3: Advancing End-to-End 3D Detection and Trackingcited by 69
  23. 2023Cross Modal Transformer: Towards Fast and Robust 3D Object Detectioncited by 68
  24. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  25. 2024MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mappingcited by 67
  26. 2022Hard to Track Objects with Irregular Motions and Similar Appearances? Make It Easier by Buffering the Matching Spacecited by 66
  27. 2023Hybrid-SORT: Weak Cues Matter for Online Multi-Object Trackingcited by 66
  28. 2023MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Trackingcited by 64
  29. 2022Unifying Short and Long-Term Tracking with Graph Hierarchiescited by 62
  30. 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
  31. 2022QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Trackingcited by 61
  32. 2024DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Predictioncited by 59
  33. 2022Accelerating DETR Convergence via Semantic-Aligned Matchingcited by 58
  34. 2024Multiple Object Tracking as ID Predictioncited by 58
  35. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  36. 2023OVTrack: Open-Vocabulary Multiple Object Trackingcited by 53
  37. 2022A Generalized Framework for Video Instance Segmentationcited by 52
  38. 2022CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusioncited by 50

50 most cited of 379 All 379 (JSON)