# Learning Multi-Object Tracking and Segmentation from Automatic Annotations

arXiv: https://arxiv.org/abs/1912.02096 · HTML: https://mlsanity.com/paper/1912.02096/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2019.

## Built on by (most cited first, 27 of 27)

- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimation](https://mlsanity.com/paper/2309.15082/) (2023)
- [Learning to Associate Every Segment for Video Panoptic Segmentation](https://mlsanity.com/paper/2106.09453/) (2021)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [TOPIC: A Parallel Association Paradigm for Multi-Object Tracking under Complex Motions and Diverse Scenes](https://mlsanity.com/paper/2308.11157/) (2023)
- [Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects](https://mlsanity.com/paper/2210.02074/) (2022)
- [OmniTracker: Unifying Object Tracking by Tracking-with-Detection](https://mlsanity.com/paper/2303.12079/) (2023)
- [Online Multi-Object Tracking with Unsupervised Re-Identification Learning and Occlusion Estimation](https://mlsanity.com/paper/2201.01297/) (2022)
- [DG-Labeler and DGL-MOTS Dataset: Boost the Autonomous Driving Perception](https://mlsanity.com/paper/2110.07790/) (2021)
- [Multi-Object Tracking and Segmentation via Neural Message Passing](https://mlsanity.com/paper/2207.07454/) (2022)
- [Multi-Object Tracking with Hallucinated and Unlabeled Videos](https://mlsanity.com/paper/2108.08836/) (2021)
- [Self-Improving Semantic Perception for Indoor Localisation](https://mlsanity.com/paper/2105.01595/) (2021)
- [Hybrid Tracker with Pixel and Instance for Video Panoptic Segmentation](https://mlsanity.com/paper/2203.01217/) (2022)
- [Camera Motion Estimation from RGB-D-Inertial Scene Flow](https://mlsanity.com/paper/2404.17251/) (2024)
- [Weakly Supervised Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2101.00667/) (2021)
- [Propagating Semantic Labels in Video Data](https://mlsanity.com/paper/2310.00783/) (2023)
- [A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects](https://mlsanity.com/paper/2506.13552/) (2025)
- [Motion Estimation for Large Displacements and Deformations](https://mlsanity.com/paper/2206.12464/) (2022)
- [Membership Inference Attack for Beluga Whales Discrimination](https://mlsanity.com/paper/2302.14769/) (2023)
- [UDA4Inst: Unsupervised Domain Adaptation for Instance Segmentation](https://mlsanity.com/paper/2405.09682/) (2024)
- [DINTR: Tracking via Diffusion-based Interpolation](https://mlsanity.com/paper/2410.10053/) (2024)
- [Learning Appearance and Motion Cues for Panoptic Tracking](https://mlsanity.com/paper/2503.09191/) (2025)
- [MOTS R-CNN: Cosine-margin-triplet loss for multi-object tracking](https://mlsanity.com/paper/2102.03512/) (2021)
- [SA-VIS: Sparse frame Annotations for training Video Instance Segmentation](https://mlsanity.com/paper/2606.20140/) (2026)

## Abstract (arXiv)

In this work we contribute a novel pipeline to automatically generate training data, and to improve over state-of-the-art multi-object tracking and segmentation (MOTS) methods. Our proposed track mining algorithm turns raw street-level videos into high-fidelity MOTS training data, is scalable and overcomes the need of expensive and time-consuming manual annotation approaches. We leverage state-of-the-art instance segmentation results in combination with optical flow predictions, also trained on automatically harvested training data. Our second major contribution is MOTSNet - a deep learning, tracking-by-detection architecture for MOTS - deploying a novel mask-pooling layer for improved object association over time. Training MOTSNet with our automatically extracted data leads to significantly improved sMOTSA scores on the novel KITTI MOTS dataset (+1.9%/+7.5% on cars/pedestrians), and MOTSNet improves by +4.1% over previously best methods on the MOTSChallenge dataset. Our most impressive finding is that we can improve over previous best-performing works, even in complete absence of manually annotated MOTS training data.