# RelationTrack: Relation-aware Multiple Object Tracking with Decoupled Representation

arXiv: https://arxiv.org/abs/2105.04322 · HTML: https://mlsanity.com/paper/2105.04322/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 77 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Joint Detection and Identification Feature Learning for Person Search](https://mlsanity.com/paper/1604.01850/) (2016)
- [Flow-Guided Feature Aggregation for Video Object Detection](https://mlsanity.com/paper/1703.10025/) (2017)
- [CityPersons: A Diverse Dataset for Pedestrian Detection](https://mlsanity.com/paper/1702.05693/) (2017)

## Built on by (most cited first, 30 of 52)

- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking](https://mlsanity.com/paper/2203.14360/) (2022)
- [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022)
- [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022)
- [MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking](https://mlsanity.com/paper/2303.10404/) (2023)
- [SparseTrack: Multi-Object Tracking by Performing Scene Decomposition based on Pseudo-Depth](https://mlsanity.com/paper/2306.05238/) (2023)
- [MOTRv3: Release-Fetch Supervision for End-to-End Multi-Object Tracking](https://mlsanity.com/paper/2305.14298/) (2023)
- [Tracking Objects as Pixel-wise Distributions](https://mlsanity.com/paper/2207.05518/) (2022)
- [Towards Discriminative Representation: Multi-view Trajectory Contrastive Learning for Online Multi-object Tracking](https://mlsanity.com/paper/2203.14208/) (2022)
- [SMILEtrack: SiMIlarity LEarning for Occlusion-Aware Multiple Object Tracking](https://mlsanity.com/paper/2211.08824/) (2022)
- [Focus On Details: Online Multi-object Tracking with Diverse Fine-grained Representation](https://mlsanity.com/paper/2302.14589/) (2023)
- [MotionTrack: Learning Motion Predictor for Multiple Object Tracking](https://mlsanity.com/paper/2306.02585/) (2023)
- [OmniTracker: Unifying Object Tracking by Tracking-with-Detection](https://mlsanity.com/paper/2303.12079/) (2023)
- [EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2402.18302/) (2024)
- [Delving into the Trajectory Long-tail Distribution for Muti-object Tracking](https://mlsanity.com/paper/2403.04700/) (2024)
- [Quality Matters: Embracing Quality Clues for Robust 3D Multi-Object Tracking](https://mlsanity.com/paper/2208.10976/) (2022)
- [Generalizing Multiple Object Tracking to Unseen Domains by Introducing Natural Language Representation](https://mlsanity.com/paper/2212.01568/) (2022)
- [Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classification](https://mlsanity.com/paper/2208.12398/) (2022)
- [Detection Recovery in Online Multi-Object Tracking with Sparse Graph Tracker](https://mlsanity.com/paper/2205.00968/) (2022)
- [In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems](https://mlsanity.com/paper/2209.04796/) (2022)
- [DeNoising-MOT: Towards Multiple Object Tracking with Severe Occlusions](https://mlsanity.com/paper/2309.04682/) (2023)
- [Cross-View Referring Multi-Object Tracking](https://mlsanity.com/paper/2412.17807/) (2024)
- [RAM: Recover Any 3D Human Motion in-the-Wild](https://mlsanity.com/paper/2603.19929/) (2026)
- [DeconfuseTrack:Dealing with Confusion for Multi-Object Tracking](https://mlsanity.com/paper/2403.02767/) (2024)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [FeatureSORT: Essential Features for Effective Tracking](https://mlsanity.com/paper/2407.04249/) (2024)
- [SimpleTrack: Rethinking and Improving the JDE Approach for Multi-Object Tracking](https://mlsanity.com/paper/2203.03985/) (2022)
- [PD-SORT: Occlusion-Robust Multi-Object Tracking Using Pseudo-Depth Cues](https://mlsanity.com/paper/2501.11288/) (2025)
- [No Train Yet Gain: Towards Generic Multi-Object Tracking in Sports and Beyond](https://mlsanity.com/paper/2506.01373/) (2025)
- [PatchTrack: Multiple Object Tracking Using Frame Patches](https://mlsanity.com/paper/2201.00080/) (2022)

## Abstract (arXiv)

Existing online multiple object tracking (MOT) algorithms often consist of two subtasks, detection and re-identification (ReID). In order to enhance the inference speed and reduce the complexity, current methods commonly integrate these double subtasks into a unified framework. Nevertheless, detection and ReID demand diverse features. This issue would result in an optimization contradiction during the training procedure. With the target of alleviating this contradiction, we devise a module named Global Context Disentangling (GCD) that decouples the learned representation into detection-specific and ReID-specific embeddings. As such, this module provides an implicit manner to balance the different requirements of these two subtasks. Moreover, we observe that preceding MOT methods typically leverage local information to associate the detected targets and neglect to consider the global semantic relation. To resolve this restriction, we develop a module, referred to as Guided Transformer Encoder (GTE), by combining the powerful reasoning ability of Transformer encoder and deformable attention. Unlike previous works, GTE avoids analyzing all the pixels and only attends to capture the relation between query nodes and a few self-adaptively selected key samples. Therefore, it is computationally efficient. Extensive experiments have been conducted on the MOT16, MOT17 and MOT20 benchmarks to demonstrate the superiority of the proposed MOT framework, namely RelationTrack. The experimental results indicate that RelationTrack has surpassed preceding methods significantly and established a new state-of-the-art performance, e.g., IDF1 of 70.5% and MOTA of 67.2% on MOT20.