# Self-Supervised Multi-Object Tracking with Cross-Input Consistency

arXiv: https://arxiv.org/abs/2111.05943 · HTML: https://mlsanity.com/paper/2111.05943/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [Multi-task Self-Supervised Visual Learning](https://mlsanity.com/paper/1708.07860/) (2017)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking](https://mlsanity.com/paper/1504.01942/) (2015)
- [MAST: A Memory-Augmented Self-supervised Tracker](https://mlsanity.com/paper/2002.07793/) (2020)
- [Joint-task Self-supervised Learning for Temporal Correspondence](https://mlsanity.com/paper/1909.11895/) (2019)
- [FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking](https://mlsanity.com/paper/1904.04989/) (2019)
- [Robust Multi-Modality Multi-Object Tracking](https://mlsanity.com/paper/1909.03850/) (2019)
- [Simple Unsupervised Multi-Object Tracking](https://mlsanity.com/paper/2006.02609/) (2020)
- [PathTrack: Fast Trajectory Annotation with Path Supervision](https://mlsanity.com/paper/1703.02437/) (2017)
- [Multi-Object Tracking with Multiple Cues and Switcher-Aware Classification](https://mlsanity.com/paper/1901.06129/) (2019)

## Built on by (most cited first, 25 of 25)

- [Simple Cues Lead to a Strong Multi-Object Tracker](https://mlsanity.com/paper/2206.04656/) (2022)
- [Self-Supervised Multi-Object Tracking with Path Consistency](https://mlsanity.com/paper/2404.05136/) (2024)
- [Object-Centric Multiple Object Tracking](https://mlsanity.com/paper/2309.00233/) (2023)
- [TrackFlow: Multi-Object Tracking with Normalizing Flows](https://mlsanity.com/paper/2308.11513/) (2023)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [Tracking without Label: Unsupervised Multiple Object Tracking via Contrastive Similarity Learning](https://mlsanity.com/paper/2309.00942/) (2023)
- [Self-Supervised Multi-Object Tracking For Autonomous Driving From Consistency Across Timescales](https://mlsanity.com/paper/2304.13147/) (2023)
- [SPAMming Labels: Efficient Annotations for the Trackers of Tomorrow](https://mlsanity.com/paper/2404.11426/) (2024)
- [Walker: Self-supervised Multiple Object Tracking by Walking on Temporal Appearance Graphs](https://mlsanity.com/paper/2409.17221/) (2024)
- [SearchTrack: Multiple Object Tracking with Object-Customized Search and Motion-Aware Features](https://mlsanity.com/paper/2210.16572/) (2022)
- [Learning a Neural Association Network for Self-supervised Multi-Object Tracking](https://mlsanity.com/paper/2411.11514/) (2024)
- [An HMM-based framework for identity-aware long-term multi-object tracking from sparse and uncertain identification: use case on long-term tracking in livestock](https://mlsanity.com/paper/2509.09962/) (2025)
- [Self-Supervised RGB-T Tracking with Cross-Input Consistency](https://mlsanity.com/paper/2301.11274/) (2023)
- [Unified Perception: Efficient Depth-Aware Video Panoptic Segmentation with Minimal Annotation Costs](https://mlsanity.com/paper/2303.01991/) (2023)
- [S3Track: Self-supervised Tracking with Soft Assignment Flow](https://mlsanity.com/paper/2305.09981/) (2023)
- [BackTrack: Robust template update via Backward Tracking of candidate template](https://mlsanity.com/paper/2308.10604/) (2023)
- [UnsMOT: Unified Framework for Unsupervised Multi-Object Tracking with Geometric Topology Guidance](https://mlsanity.com/paper/2309.01078/) (2023)
- [Collecting Consistently High Quality Object Tracks with Minimal Human Involvement by Using Self-Supervised Learning to Detect Tracker Errors](https://mlsanity.com/paper/2405.03643/) (2024)
- [Self-Supervised Partial Cycle-Consistency for Multi-View Matching](https://mlsanity.com/paper/2501.06000/) (2025)
- [SAM2Auto: Auto Annotation Using FLASH](https://mlsanity.com/paper/2506.07850/) (2025)
- [Multi-animal tracking in Transition: Comparative Insights into Established and Emerging Methods](https://mlsanity.com/paper/2509.11873/) (2025)
- [SSVIF: Self-Supervised Segmentation-Oriented Visible and Infrared Image Fusion](https://mlsanity.com/paper/2509.22450/) (2025)
- [CalibFree: Self-Supervised View Feature Separation for Calibration-Free Multi-Camera Multi-Object Tracking](https://mlsanity.com/paper/2605.09245/) (2026)
- [Cycle Consistency in Video Object-Centric Learning](https://mlsanity.com/paper/2605.30211/) (2026)
- [Beyond Appearance: A Multi-cue Framework and Large-scale Benchmark for Pedestrian Association and Tracking on Mobile Aerial-Ground Platforms](https://mlsanity.com/paper/2607.23803/) (2026)

## Abstract (arXiv)

In this paper, we propose a self-supervised learning procedure for training a robust multi-object tracking (MOT) model given only unlabeled video. While several self-supervisory learning signals have been proposed in prior work on single-object tracking, such as color propagation and cycle-consistency, these signals cannot be directly applied for training RNN models, which are needed to achieve accurate MOT: they yield degenerate models that, for instance, always match new detections to tracks with the closest initial detections. We propose a novel self-supervisory signal that we call cross-input consistency: we construct two distinct inputs for the same sequence of video, by hiding different information about the sequence in each input. We then compute tracks in that sequence by applying an RNN model independently on each input, and train the model to produce consistent tracks across the two inputs. We evaluate our unsupervised method on MOT17 and KITTI -- remarkably, we find that, despite training only on unlabeled video, our unsupervised approach outperforms four supervised methods published in the last 1--2 years, including Tracktor++, FAMNet, GSM, and mmMOT.