# ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation

arXiv: https://arxiv.org/abs/2007.03200 · HTML: https://mlsanity.com/paper/2007.03200/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 17 of 17)

- [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022)
- [CML-MOTS: Collaborative Multi-task Learning for Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2311.00987/) (2023)
- [Track Targets by Dense Spatio-Temporal Position Encoding](https://mlsanity.com/paper/2210.09455/) (2022)
- [Z-GMOT: Zero-shot Generic Multiple Object Tracking](https://mlsanity.com/paper/2305.17648/) (2023)
- [A Unified Multi-view Multi-person Tracking Framework](https://mlsanity.com/paper/2302.03820/) (2023)
- [Walker: Self-supervised Multiple Object Tracking by Walking on Temporal Appearance Graphs](https://mlsanity.com/paper/2409.17221/) (2024)
- [Seg2Track-SAM2: SAM2-based Multi-object Tracking and Segmentation](https://mlsanity.com/paper/2509.11772/) (2025)
- [SearchTrack: Multiple Object Tracking with Object-Customized Search and Motion-Aware Features](https://mlsanity.com/paper/2210.16572/) (2022)
- [The Second-place Solution for CVPR 2022 SoccerNet Tracking Challenge](https://mlsanity.com/paper/2211.13481/) (2022)
- [Multi-Object Tracking and Segmentation with a Space-Time Memory Network](https://mlsanity.com/paper/2110.11284/) (2021)
- [The Second-place Solution for ECCV 2022 Multiple People Tracking in Group Dance Challenge](https://mlsanity.com/paper/2211.13509/) (2022)
- [SSGA-Net: Stepwise Spatial Global-local Aggregation Networks for for Autonomous Driving](https://mlsanity.com/paper/2405.18857/) (2024)
- [MOTS R-CNN: Cosine-margin-triplet loss for multi-object tracking](https://mlsanity.com/paper/2102.03512/) (2021)
- [MeNToS: Tracklets Association with a Space-Time Memory Network](https://mlsanity.com/paper/2107.07067/) (2021)
- [PolyTrack: Tracking with Bounding Polygons](https://mlsanity.com/paper/2111.01606/) (2021)
- [Actor-identified Spatiotemporal Action Detection -- Detecting Who Is Doing What in Videos](https://mlsanity.com/paper/2208.12940/) (2022)
- [Seg2Track++: Probabilistic Track Validation and Data Association for Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2606.03875/) (2026)

## Abstract (arXiv)

We aim to improve the performance of Multiple Object Tracking and Segmentation (MOTS) by refinement. However, it remains challenging for refining MOTS results, which could be attributed to that appearance features are not adapted to target videos and it is also difficult to find proper thresholds to discriminate them. To tackle this issue, we propose a self-supervised refining MOTS (i.e., ReMOTS) framework. ReMOTS mainly takes four steps to refine MOTS results from the data association perspective. (1) Training the appearance encoder using predicted masks. (2) Associating observations across adjacent frames to form short-term tracklets. (3) Training the appearance encoder using short-term tracklets as reliable pseudo labels. (4) Merging short-term tracklets to long-term tracklets utilizing adopted appearance features and thresholds that are automatically obtained from statistical information. Using ReMOTS, we reached the 1st place on CVPR 2020 MOTS Challenge 1, with an sMOTSA score of 69.9.