# Unifying Short and Long-Term Tracking with Graph Hierarchies

arXiv: https://arxiv.org/abs/2212.03038 · HTML: https://mlsanity.com/paper/2212.03038/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 77 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion](https://mlsanity.com/paper/2111.14690/) (2021)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking](https://mlsanity.com/paper/2010.07548/) (2020)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [MeMOT: Multi-Object Tracking with Memory](https://mlsanity.com/paper/2203.16761/) (2022)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)
- [Towards Grand Unification of Object Tracking](https://mlsanity.com/paper/2207.07078/) (2022)

## Built on by (most cited first, 30 of 62)

- [Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking](https://mlsanity.com/paper/2308.00783/) (2023)
- [Matching Anything by Segmenting Anything](https://mlsanity.com/paper/2406.04221/) (2024)
- [SparseTrack: Multi-Object Tracking by Performing Scene Decomposition based on Pseudo-Depth](https://mlsanity.com/paper/2306.05238/) (2023)
- [UCMCTrack: Multi-Object Tracking with Uniform Camera Motion Compensation](https://mlsanity.com/paper/2312.08952/) (2023)
- [HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding](https://mlsanity.com/paper/2312.03050/) (2023)
- [SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation](https://mlsanity.com/paper/2504.04519/) (2025)
- [BoostTrack++: using tracklet information to detect more objects in multiple object tracking](https://mlsanity.com/paper/2408.13003/) (2024)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [MCBLT: Multi-Camera Multi-Object 3D Tracking in Long Videos](https://mlsanity.com/paper/2412.00692/) (2024)
- [Learning Data Association for Multi-Object Tracking using Only Coordinates](https://mlsanity.com/paper/2403.08018/) (2024)
- [Z-GMOT: Zero-shot Generic Multiple Object Tracking](https://mlsanity.com/paper/2305.17648/) (2023)
- [Multi-Scene Generalized Trajectory Global Graph Solver with Composite Nodes for Multiple Object Tracking](https://mlsanity.com/paper/2312.08951/) (2023)
- [SPAMming Labels: Efficient Annotations for the Trackers of Tomorrow](https://mlsanity.com/paper/2404.11426/) (2024)
- [SLAck: Semantic, Location, and Appearance Aware Open-Vocabulary Tracking](https://mlsanity.com/paper/2409.11235/) (2024)
- [Contrastive Learning for Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2311.08043/) (2023)
- [No Train Yet Gain: Towards Generic Multi-Object Tracking in Sports and Beyond](https://mlsanity.com/paper/2506.01373/) (2025)
- [Associate Everything Detected: Facilitating Tracking-by-Detection to the Unknown](https://mlsanity.com/paper/2409.09293/) (2024)
- [MVTrajecter: Multi-View Pedestrian Tracking with Trajectory Motion Cost and Trajectory Appearance Cost](https://mlsanity.com/paper/2509.01157/) (2025)
- [SegMASt3R: Geometry Grounded Segment Matching](https://mlsanity.com/paper/2510.05051/) (2025)
- [UncertaintyTrack: Exploiting Detection and Localization Uncertainty in Multi-Object Tracking](https://mlsanity.com/paper/2402.12303/) (2024)
- [Engineering an Efficient Object Tracker for Non-Linear Motion](https://mlsanity.com/paper/2407.00738/) (2024)
- [Deep Learning-Based Multi-Object Tracking: A Comprehensive Survey from Foundations to State-of-the-Art](https://mlsanity.com/paper/2506.13457/) (2025)
- [XS-VID: A Large-Scale Benchmark for Small Object Detection and Tracking in Videos](https://mlsanity.com/paper/2407.18137/) (2024)
- [NOOUGAT: Towards Unified Online and Offline Multi-Object Tracking](https://mlsanity.com/paper/2509.02111/) (2025)
- [DecoderTracker: Decoder-Only Method for Multiple-Object Tracking](https://mlsanity.com/paper/2310.17170/) (2023)
- [SFSORT: Scene Features-based Simple Online Real-Time Tracker](https://mlsanity.com/paper/2404.07553/) (2024)
- [Is a Pure Transformer Effective for Separated and Online Multi-Object Tracking?](https://mlsanity.com/paper/2405.14119/) (2024)
- [Multi-Granularity Language-Guided Training for Multi-Object Tracking](https://mlsanity.com/paper/2406.04844/) (2024)
- [EASE: Embodied Active Event Perception via Self-Supervised Energy Minimization](https://mlsanity.com/paper/2506.17516/) (2025)
- [One Graph to Track Them All: Dynamic GNNs for Single- and Multi-View Tracking](https://mlsanity.com/paper/2507.08494/) (2025)

## Abstract (arXiv)

Tracking objects over long videos effectively means solving a spectrum of problems, from short-term association for un-occluded objects to long-term association for objects that are occluded and then reappear in the scene. Methods tackling these two tasks are often disjoint and crafted for specific scenarios, and top-performing approaches are often a mix of techniques, which yields engineering-heavy solutions that lack generality. In this work, we question the need for hybrid approaches and introduce SUSHI, a unified and scalable multi-object tracker. Our approach processes long clips by splitting them into a hierarchy of subclips, which enables high scalability. We leverage graph neural networks to process all levels of the hierarchy, which makes our model unified across temporal scales and highly general. As a result, we obtain significant improvements over state-of-the-art on four diverse datasets. Our code and models are available at bit.ly/sushi-mot.