# Uncertainty-aware Unsupervised Multi-Object Tracking

arXiv: https://arxiv.org/abs/2307.15409 · HTML: https://mlsanity.com/paper/2307.15409/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 61 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Builds on (most cited first, 30 of 45)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Detection and Tracking Meet Drones Challenge](https://mlsanity.com/paper/2001.06303/) (2020)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [Deep Image Homography Estimation](https://mlsanity.com/paper/1606.03798/) (2016)
- [Can multi-label classification networks know what they don't know?](https://mlsanity.com/paper/2109.14162/) (2021)
- [Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification](https://mlsanity.com/paper/1809.04427/) (2018)

## Built on by (most cited first, 12 of 12)

- [Self-Supervised Multi-Object Tracking with Path Consistency](https://mlsanity.com/paper/2404.05136/) (2024)
- [RingMo-Aerial: An Aerial Remote Sensing Foundation Model With Affine Transformation Contrastive Learning](https://mlsanity.com/paper/2409.13366/) (2024)
- [View-Centric Multi-Object Tracking with Homographic Matching in Moving UAV](https://mlsanity.com/paper/2403.10830/) (2024)
- [SPAMming Labels: Efficient Annotations for the Trackers of Tomorrow](https://mlsanity.com/paper/2404.11426/) (2024)
- [RGB-Event based Pedestrian Attribute Recognition: A Benchmark Dataset and An Asymmetric RWKV Fusion Framework](https://mlsanity.com/paper/2504.10018/) (2025)
- [DroneMOT: Drone-based Multi-Object Tracking Considering Detection Difficulties and Simultaneous Moving of Drones and Objects](https://mlsanity.com/paper/2407.09051/) (2024)
- [Learning a Neural Association Network for Self-supervised Multi-Object Tracking](https://mlsanity.com/paper/2411.11514/) (2024)
- [Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos](https://mlsanity.com/paper/2508.01730/) (2025)
- [A General Approach to Visualizing Uncertainty in Statistical Graphics](https://mlsanity.com/paper/2508.00937/) (2025)
- [SocialTrack: Multi-Object Tracking in Complex Urban Traffic Scenes Inspired by Social Behavior](https://mlsanity.com/paper/2508.12777/) (2025)
- [Breaking Smooth-Motion Assumptions: A UAV Benchmark for Multi-Object Tracking in Complex and Adverse Conditions](https://mlsanity.com/paper/2603.05970/) (2026)
- [Attention Is not Everything: Efficient Alternatives for Vision](https://mlsanity.com/paper/2604.17439/) (2026)

## Abstract (arXiv)

Without manually annotated identities, unsupervised multi-object trackers are inferior to learning reliable feature embeddings. It causes the similarity-based inter-frame association stage also be error-prone, where an uncertainty problem arises. The frame-by-frame accumulated uncertainty prevents trackers from learning the consistent feature embedding against time variation. To avoid this uncertainty problem, recent self-supervised techniques are adopted, whereas they failed to capture temporal relations. The interframe uncertainty still exists. In fact, this paper argues that though the uncertainty problem is inevitable, it is possible to leverage the uncertainty itself to improve the learned consistency in turn. Specifically, an uncertainty-based metric is developed to verify and rectify the risky associations. The resulting accurate pseudo-tracklets boost learning the feature consistency. And accurate tracklets can incorporate temporal information into spatial transformation. This paper proposes a tracklet-guided augmentation strategy to simulate tracklets' motion, which adopts a hierarchical uncertainty-based sampling mechanism for hard sample mining. The ultimate unsupervised MOT framework, namely U2MOT, is proven effective on MOT-Challenges and VisDrone-MOT benchmark. U2MOT achieves a SOTA performance among the published supervised and unsupervised trackers.