# STURE: Spatial-Temporal Mutual Representation Learning for Robust Data Association in Online Multi-Object Tracking

arXiv: https://arxiv.org/abs/2201.06824 · HTML: https://mlsanity.com/paper/2201.06824/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 66 references that are in the map.
- First posted on arXiv 18 Jan 2022.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)](https://mlsanity.com/paper/1711.09349/) (2017)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking](https://mlsanity.com/paper/1511.04136/) (2015)
- [Learning by tracking: Siamese CNN for robust target association](https://mlsanity.com/paper/1604.07866/) (2016)
- [Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies](https://mlsanity.com/paper/1701.01909/) (2017)
- [Spatial-Temporal Relation Networks for Multi-Object Tracking](https://mlsanity.com/paper/1904.11489/) (2019)
- [FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking](https://mlsanity.com/paper/1904.04989/) (2019)
- [AlignedReID: Surpassing Human-Level Performance in Person Re-Identification](https://mlsanity.com/paper/1711.08184/) (2017)
- [DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer](https://mlsanity.com/paper/1707.01220/) (2017)
- [Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanism](https://mlsanity.com/paper/1708.02843/) (2017)
- [A Unified Object Motion and Affinity Model for Online Multi-Object Tracking](https://mlsanity.com/paper/2003.11291/) (2020)
- [CVPR19 Tracking and Detection Challenge: How crowded can it get?](https://mlsanity.com/paper/1906.04567/) (2019)
- [Discriminative Appearance Modeling with Multi-track Pooling for Real-time Multi-object Tracking](https://mlsanity.com/paper/2101.12159/) (2021)
- [Temporal Knowledge Propagation for Image-to-Video Person Re-identification](https://mlsanity.com/paper/1908.03885/) (2019)
- [Multi-Object Tracking with Multiple Cues and Switcher-Aware Classification](https://mlsanity.com/paper/1901.06129/) (2019)
- [Occlusion-Robust Online Multi-Object Visual Tracking using a GM-PHD Filter with CNN-Based Re-Identification](https://mlsanity.com/paper/1912.05949/) (2019)
- [Top-push Video-based Person Re-identification](https://mlsanity.com/paper/1604.08683/) (2016)

## Abstract (arXiv)

Online multi-object tracking (MOT) is a longstanding task for computer vision and intelligent vehicle platform. At present, the main paradigm is tracking-by-detection, and the main difficulty of this paradigm is how to associate current candidate detections with historical tracklets. However, in the MOT scenarios, each historical tracklet is composed of an object sequence, while each candidate detection is just a flat image, which lacks temporal features of the object sequence. The feature difference between current candidate detections and historical tracklets makes the object association much harder. Therefore, we propose a Spatial-Temporal Mutual Representation Learning (STURE) approach which learns spatial-temporal representations between current candidate detections and historical sequences in a mutual representation space. For historical trackelets, the detection learning network is forced to match the representations of sequence learning network in a mutual representation space. The proposed approach is capable of extracting more distinguishing detection and sequence representations by using various designed losses in object association. As a result, spatial-temporal feature is learned mutually to reinforce the current detection features, and the feature difference can be relieved. To prove the robustness of the STURE, it is applied to the public MOT challenge benchmarks and performs well compared with various state-of-the-art online MOT trackers based on identity-preserving metrics.