# Bridging the Gap Between End-to-end and Non-End-to-end Multi-Object Tracking

arXiv: https://arxiv.org/abs/2305.12724 · HTML: https://mlsanity.com/paper/2305.12724/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 49 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 42)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022)
- [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion](https://mlsanity.com/paper/2111.14690/) (2021)
- [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022)
- [Harmonious Attention Network for Person Re-Identification](https://mlsanity.com/paper/1802.08122/) (2018)

## Built on by (most cited first, 20 of 20)

- [SoccerNet 2023 Challenges Results](https://mlsanity.com/paper/2309.06006/) (2023)
- [EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2402.18302/) (2024)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2410.13437/) (2024)
- [Cognitive Disentanglement for Referring Multi-Object Tracking](https://mlsanity.com/paper/2503.11496/) (2025)
- [CAMELTrack: Context-Aware Multi-cue ExpLoitation for Online Multi-Object Tracking](https://mlsanity.com/paper/2505.01257/) (2025)
- [DecoderTracker: Decoder-Only Method for Multiple-Object Tracking](https://mlsanity.com/paper/2310.17170/) (2023)
- [Is a Pure Transformer Effective for Separated and Online Multi-Object Tracking?](https://mlsanity.com/paper/2405.14119/) (2024)
- [Tell Me What to Track: Infusing Robust Language Guidance for Enhanced Referring Multi-Object Tracking](https://mlsanity.com/paper/2412.12561/) (2024)
- [AR-MOT: Autoregressive Multi-object Tracking](https://mlsanity.com/paper/2601.01925/) (2026)
- [FastTrackTr:Towards Fast Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2411.15811/) (2024)
- [To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software](https://mlsanity.com/paper/2508.20892/) (2025)
- [Motion-Aware Transformer for Multi-Object Tracking](https://mlsanity.com/paper/2509.21715/) (2025)
- [From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking](https://mlsanity.com/paper/2512.02392/) (2025)
- [Tracking by Detection and Query: An Efficient End-to-End Framework for Multi-Object Tracking](https://mlsanity.com/paper/2411.06197/) (2024)
- [VSE-MOT: Multi-Object Tracking in Low-Quality Video Scenes Guided by Visual Semantic Enhancement](https://mlsanity.com/paper/2509.14060/) (2025)
- [IMPose: Interactive Multi-person Pose Estimation with Dynamic Correction Propagation](https://mlsanity.com/paper/2606.04480/) (2026)
- [Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking](https://mlsanity.com/paper/2609.00924/) (2026)
- [LiAM-SAM: Lifecycle-Aware Memory for Robust SAM2-Based MOT](https://mlsanity.com/paper/2609.28078/) (2026)

## Abstract (arXiv)

Existing end-to-end Multi-Object Tracking (e2e-MOT) methods have not surpassed non-end-to-end tracking-by-detection methods. One potential reason is its label assignment strategy during training that consistently binds the tracked objects with tracking queries and then assigns the few newborns to detection queries. With one-to-one bipartite matching, such an assignment will yield unbalanced training, i.e., scarce positive samples for detection queries, especially for an enclosed scene, as the majority of the newborns come on stage at the beginning of videos. Thus, e2e-MOT will be easier to yield a tracking terminal without renewal or re-initialization, compared to other tracking-by-detection methods. To alleviate this problem, we present Co-MOT, a simple and effective method to facilitate e2e-MOT by a novel coopetition label assignment with a shadow concept. Specifically, we add tracked objects to the matching targets for detection queries when performing the label assignment for training the intermediate decoders. For query initialization, we expand each query by a set of shadow counterparts with limited disturbance to itself. With extensive ablations, Co-MOT achieves superior performance without extra costs, e.g., 69.4% HOTA on DanceTrack and 52.8% TETA on BDD100K. Impressively, Co-MOT only requires 38% FLOPs of MOTRv2 to attain a similar performance, resulting in the 1.4× faster inference speed.