# Learning a Spatio-Temporal Embedding for Video Instance Segmentation

arXiv: https://arxiv.org/abs/1912.08969 · HTML: https://mlsanity.com/paper/1912.08969/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2019.

## Built on by (most cited first, 7 of 7)

- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Weakly Supervised Instance Segmentation for Videos with Temporal Mask Consistency](https://mlsanity.com/paper/2103.12886/) (2021)
- [RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation](https://mlsanity.com/paper/2203.04187/) (2022)
- [HOPE: Hierarchical Spatial-temporal Network for Occupancy Flow Prediction](https://mlsanity.com/paper/2206.10118/) (2022)
- [VideoClick: Video Object Segmentation with a Single Click](https://mlsanity.com/paper/2101.06545/) (2021)
- [Instance Segmentation with Cross-Modal Consistency](https://mlsanity.com/paper/2210.08113/) (2022)
- [Robust semi-automatic vessel tracing in the human retinal image by an instance segmentation neural network](https://mlsanity.com/paper/2402.10055/) (2024)

## Abstract (arXiv)

We present a novel embedding approach for video instance segmentation. Our method learns a spatio-temporal embedding integrating cues from appearance, motion, and geometry; a 3D causal convolutional network models motion, and a monocular self-supervised depth loss models geometry. In this embedding space, video-pixels of the same instance are clustered together while being separated from other instances, to naturally track instances over time without any complex post-processing. Our network runs in real-time as our architecture is entirely causal - we do not incorporate information from future frames, contrary to previous methods. We show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset.