# Spatial-Temporal Relation Networks for Multi-Object Tracking

arXiv: https://arxiv.org/abs/1904.11489 · HTML: https://mlsanity.com/paper/1904.11489/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2019.

## Built on by (most cited first, 30 of 74)

- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [Occluded Video Instance Segmentation: A Benchmark](https://mlsanity.com/paper/2102.01558/) (2021)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)
- [SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation](https://mlsanity.com/paper/2103.10284/) (2021)
- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [Unifying Short and Long-Term Tracking with Graph Hierarchies](https://mlsanity.com/paper/2212.03038/) (2022)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [DEFT: Detection Embeddings for Tracking](https://mlsanity.com/paper/2102.02267/) (2021)
- [Simple Cues Lead to a Strong Multi-Object Tracker](https://mlsanity.com/paper/2206.04656/) (2022)
- [Learning a Proposal Classifier for Multiple Object Tracking](https://mlsanity.com/paper/2103.07889/) (2021)
- [AnimalTrack: A Benchmark for Multi-Animal Tracking in the Wild](https://mlsanity.com/paper/2205.00158/) (2022)
- [Time3D: End-to-End Joint Monocular 3D Object Detection and Tracking for Autonomous Driving](https://mlsanity.com/paper/2205.14882/) (2022)
- [One More Check: Making "Fake Background" Be Tracked Again](https://mlsanity.com/paper/2104.09441/) (2021)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [Discriminative Appearance Modeling with Multi-track Pooling for Real-time Multi-object Tracking](https://mlsanity.com/paper/2101.12159/) (2021)
- [Tracking People with 3D Representations](https://mlsanity.com/paper/2111.07868/) (2021)
- [Looking Beyond Two Frames: End-to-End Multi-Object Tracking Using Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.14829/) (2021)
- [ByteTrackV2: 2D and 3D Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2303.15334/) (2023)
- [Making Higher Order MOT Scalable: An Efficient Approximate Solver for Lifted Disjoint Paths](https://mlsanity.com/paper/2108.10606/) (2021)
- [EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2402.18302/) (2024)
- [Object Permanence Emerges in a Random Walk along Memory](https://mlsanity.com/paper/2204.01784/) (2022)
- [Split and Connect: A Universal Tracklet Booster for Multi-Object Tracking](https://mlsanity.com/paper/2105.02426/) (2021)
- [Online Multi-Object Tracking with Unsupervised Re-Identification Learning and Occlusion Estimation](https://mlsanity.com/paper/2201.01297/) (2022)
- [EmbedTrack -- Simultaneous Cell Segmentation and Tracking Through Learning Offsets and Clustering Bandwidths](https://mlsanity.com/paper/2204.10713/) (2022)
- [Tracking People by Predicting 3D Appearance, Location & Pose](https://mlsanity.com/paper/2112.04477/) (2021)
- [Detection Recovery in Online Multi-Object Tracking with Sparse Graph Tracker](https://mlsanity.com/paper/2205.00968/) (2022)
- [DynamicDet: A Unified Dynamic Architecture for Object Detection](https://mlsanity.com/paper/2304.05552/) (2023)

## Abstract (arXiv)

Recent progress in multiple object tracking (MOT) has shown that a robust similarity score is key to the success of trackers. A good similarity score is expected to reflect multiple cues, e.g. appearance, location, and topology, over a long period of time. However, these cues are heterogeneous, making them hard to be combined in a unified network. As a result, existing methods usually encode them in separate networks or require a complex training approach. In this paper, we present a unified framework for similarity measurement which could simultaneously encode various cues and perform reasoning across both spatial and temporal domains. We also study the feature representation of a tracklet-object pair in depth, showing a proper design of the pair features can well empower the trackers. The resulting approach is named spatial-temporal relation networks (STRN). It runs in a feed-forward way and can be trained in an end-to-end manner. The state-of-the-art accuracy was achieved on all of the MOT15-17 benchmarks using public detection and online settings.