# Siamese-DETR for Generic Multi-Object Tracking arXiv: https://arxiv.org/abs/2310.17875 ยท HTML: https://mlsanity.com/paper/2310.17875/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 47 of its 74 references that are in the map. - First posted on arXiv 27 Oct 2023. ## Builds on (most cited first, 30 of 47) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018) - [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022) - [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019) - [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021) - [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017) - [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016) - [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021) - [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018) - [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014) - [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022) - [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016) - [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021) - [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022) - [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020) - [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018) - [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021) - [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021) - [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019) - [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022) - [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021) ## Built on by (most cited first, 11 of 11) - [OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving](https://mlsanity.com/paper/2405.20337/) (2024) - [Frequency Dynamic Convolution for Dense Image Prediction](https://mlsanity.com/paper/2503.18783/) (2025) - [EventHDR: from Event to High-Speed HDR Videos and Beyond](https://mlsanity.com/paper/2409.17029/) (2024) - [Distilling Monocular Foundation Model for Fine-grained Depth Completion](https://mlsanity.com/paper/2503.16970/) (2025) - [Visible-Thermal Multiple Object Tracking: Large-scale Video Dataset and Progressive Fusion Approach](https://mlsanity.com/paper/2408.00969/) (2024) - [POPCat: Propagation of particles for complex annotation tasks](https://mlsanity.com/paper/2406.17183/) (2024) - [PR-DETR: Injecting Position and Relation Prior for Dense Video Captioning](https://mlsanity.com/paper/2506.16082/) (2025) - [Multi-tracklet Tracking for Generic Targets with Adaptive Detection Clustering](https://mlsanity.com/paper/2508.05172/) (2025) - [SatSAM2: Motion-Constrained Video Object Tracking in Satellite Imagery using Promptable SAM2 and Kalman Priors](https://mlsanity.com/paper/2511.18264/) (2025) - [Reference-based Category Discovery: Unsupervised Object Detection with Category Awareness](https://mlsanity.com/paper/2605.04606/) (2026) - [UGO: Unified Architecture for General Multi-Object Tracking by Segmentation](https://mlsanity.com/paper/2609.37339/) (2026) ## Abstract (arXiv) The ability to detect and track the dynamic objects in different scenes is fundamental to real-world applications, e.g., autonomous driving and robot navigation. However, traditional Multi-Object Tracking (MOT) is limited to tracking objects belonging to the pre-defined closed-set categories. Recently, Open-Vocabulary MOT (OVMOT) and Generic MOT (GMOT) are proposed to track interested objects beyond pre-defined categories with the given text prompt and template image. However, the expensive well pre-trained (vision-)language model and fine-grained category annotations are required to train OVMOT models. In this paper, we focus on GMOT and propose a simple but effective method, Siamese-DETR, for GMOT. Only the commonly used detection datasets (e.g., COCO) are required for training. Different from existing GMOT methods, which train a Single Object Tracking (SOT) based detector to detect interested objects and then apply a data association based MOT tracker to get the trajectories, we leverage the inherent object queries in DETR variants. Specifically: 1) The multi-scale object queries are designed based on the given template image, which are effective for detecting different scales of objects with the same category as the template image; 2) A dynamic matching training strategy is introduced to train Siamese-DETR on commonly used detection datasets, which takes full advantage of provided annotations; 3) The online tracking pipeline is simplified through a tracking-by-query manner by incorporating the tracked boxes in previous frame as additional query boxes. The complex data association is replaced with the much simpler Non-Maximum Suppression (NMS). Extensive experimental results show that Siamese-DETR surpasses existing MOT methods on GMOT-40 dataset by a large margin. Codes are avaliable at urlhttps://github.com/yumu-173/Siamese-DETR.