# Towards Discriminative Representation: Multi-view Trajectory Contrastive Learning for Online Multi-object Tracking

arXiv: https://arxiv.org/abs/2203.14208 · HTML: https://mlsanity.com/paper/2203.14208/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 54 references that are in the map.
- First posted on arXiv 27 Mar 2022.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking](https://mlsanity.com/paper/1504.01942/) (2015)
- [Detect to Track and Track to Detect](https://mlsanity.com/paper/1710.03958/) (2017)
- [Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking](https://mlsanity.com/paper/2007.14557/) (2020)

## Built on by (most cited first, 30 of 33)

- [MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking](https://mlsanity.com/paper/2303.10404/) (2023)
- [CTVIS: Consistent Training for Online Video Instance Segmentation](https://mlsanity.com/paper/2307.12616/) (2023)
- [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022)
- [Simple Cues Lead to a Strong Multi-Object Tracker](https://mlsanity.com/paper/2206.04656/) (2022)
- [MOTRv3: Release-Fetch Supervision for End-to-End Multi-Object Tracking](https://mlsanity.com/paper/2305.14298/) (2023)
- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [FarSight: A Physics-Driven Whole-Body Biometric System at Large Distance and Altitude](https://mlsanity.com/paper/2306.17206/) (2023)
- [Focus On Details: Online Multi-object Tracking with Diverse Fine-grained Representation](https://mlsanity.com/paper/2302.14589/) (2023)
- [Towards Generalizable Multi-Object Tracking](https://mlsanity.com/paper/2406.00429/) (2024)
- [EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2402.18302/) (2024)
- [Delving into the Trajectory Long-tail Distribution for Muti-object Tracking](https://mlsanity.com/paper/2403.04700/) (2024)
- [Quality Matters: Embracing Quality Clues for Robust 3D Multi-Object Tracking](https://mlsanity.com/paper/2208.10976/) (2022)
- [Generalizing Multiple Object Tracking to Unseen Domains by Introducing Natural Language Representation](https://mlsanity.com/paper/2212.01568/) (2022)
- [Self-Supervised Multi-Object Tracking with Path Consistency](https://mlsanity.com/paper/2404.05136/) (2024)
- [Uncertainty-aware Unsupervised Multi-Object Tracking](https://mlsanity.com/paper/2307.15409/) (2023)
- [RAM: Recover Any 3D Human Motion in-the-Wild](https://mlsanity.com/paper/2603.19929/) (2026)
- [OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2503.10616/) (2025)
- [Collaborative Tracking Learning for Frame-Rate-Insensitive Multi-Object Tracking](https://mlsanity.com/paper/2308.05911/) (2023)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [Tracking without Label: Unsupervised Multiple Object Tracking via Contrastive Similarity Learning](https://mlsanity.com/paper/2309.00942/) (2023)
- [Contrastive Learning for Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2311.08043/) (2023)
- [MetaViewer: Towards A Unified Multi-View Representation](https://mlsanity.com/paper/2303.06329/) (2023)
- [Single-Shot and Multi-Shot Feature Learning for Multi-Object Tracking](https://mlsanity.com/paper/2311.10382/) (2023)
- [Neural Multimodal Topic Modeling: A Comprehensive Evaluation](https://mlsanity.com/paper/2403.17308/) (2024)
- [MVTrajecter: Multi-View Pedestrian Tracking with Trajectory Motion Cost and Trajectory Appearance Cost](https://mlsanity.com/paper/2509.01157/) (2025)
- [Handling Heavy Occlusion in Dense Crowd Tracking by Focusing on the Heads](https://mlsanity.com/paper/2304.07705/) (2023)
- [Representation Alignment Contrastive Regularization for Multi-Object Tracking](https://mlsanity.com/paper/2404.02562/) (2024)
- [IP-MOT: Instance Prompt Learning for Cross-Domain Multi-Object Tracking](https://mlsanity.com/paper/2410.23907/) (2024)
- [Multi Camera Connected Vision System with Multi View Analytics: A Comprehensive Survey](https://mlsanity.com/paper/2510.09731/) (2025)
- [From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking](https://mlsanity.com/paper/2512.02392/) (2025)

## Abstract (arXiv)

Discriminative representation is crucial for the association step in multi-object tracking. Recent work mainly utilizes features in single or neighboring frames for constructing metric loss and empowering networks to extract representation of targets. Although this strategy is effective, it fails to fully exploit the information contained in a whole trajectory. To this end, we propose a strategy, namely multi-view trajectory contrastive learning, in which each trajectory is represented as a center vector. By maintaining all the vectors in a dynamically updated memory bank, a trajectory-level contrastive loss is devised to explore the inter-frame information in the whole trajectories. Besides, in this strategy, each target is represented as multiple adaptively selected keypoints rather than a pre-defined anchor or center. This design allows the network to generate richer representation from multiple views of the same target, which can better characterize occluded objects. Additionally, in the inference stage, a similarity-guided feature fusion strategy is developed for further boosting the quality of the trajectory representation. Extensive experiments have been conducted on MOTChallenge to verify the effectiveness of the proposed techniques. The experimental results indicate that our method has surpassed preceding trackers and established new state-of-the-art performance.