# MOTS R-CNN: Cosine-margin-triplet loss for multi-object tracking

arXiv: https://arxiv.org/abs/2102.03512 · HTML: https://mlsanity.com/paper/2102.03512/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 6 Feb 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Harmonious Attention Network for Person Re-Identification](https://mlsanity.com/paper/1802.08122/) (2018)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification](https://mlsanity.com/paper/1809.04427/) (2018)
- [Online Multi-Object Tracking with Dual Matching Attention Networks](https://mlsanity.com/paper/1902.00749/) (2019)
- [Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies](https://mlsanity.com/paper/1701.01909/) (2017)
- [POI: Multiple Object Tracking with High Performance Detection and Appearance Feature](https://mlsanity.com/paper/1610.06136/) (2016)
- [The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction](https://mlsanity.com/paper/1912.06445/) (2019)
- [Track to Reconstruct and Reconstruct to Track](https://mlsanity.com/paper/1910.00130/) (2019)
- [Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor](https://mlsanity.com/paper/1504.02340/) (2015)

## Abstract (arXiv)

One of the central tasks of multi-object tracking involves learning a distance metric that is consistent with the semantic similarities of objects. The design of an appropriate loss function that encourages discriminative feature learning is among the most crucial challenges in deep neural network-based metric learning. Despite significant progress, slow convergence and a poor local optimum of the existing contrastive and triplet loss based deep metric learning methods necessitates a better solution. In this paper, we propose cosine-margin-contrastive (CMC) and cosine-margin-triplet (CMT) loss by reformulating both contrastive and triplet loss functions from the perspective of cosine distance. The proposed reformulation as a cosine loss is achieved by feature normalization which distributes the learned features on a hypersphere. We then propose the MOTS R-CNN framework for joint multi-object tracking and segmentation, particularly targeted at improving the tracking performance. Specifically, the tracking problem is addressed through deep metric learning based on the proposed loss functions. We propose a scale-invariant tracking by using a multi-layer feature aggregation scheme to make the model robust against object scale variations and occlusions. The MOTS R-CNN achieves the state-of-the-art tracking performance on the KITTI MOTS dataset. We show that the MOTS R-CNN reduces the identity switching by 62% and 61% on cars and pedestrians, respectively in comparison to Track R-CNN.