# Spatiotemporal CNN for Video Object Segmentation

arXiv: https://arxiv.org/abs/1904.02363 · HTML: https://mlsanity.com/paper/1904.02363/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2019.

## Built on by (most cited first, 24 of 24)

- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [LVOS: A Benchmark for Long-term Video Object Segmentation](https://mlsanity.com/paper/2211.10181/) (2022)
- [SwiftNet: Real-time Video Object Segmentation](https://mlsanity.com/paper/2102.04604/) (2021)
- [LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation](https://mlsanity.com/paper/2404.19326/) (2024)
- [Tackling Background Distraction in Video Object Segmentation](https://mlsanity.com/paper/2207.06953/) (2022)
- [Convolutional Neural Networks on Manifolds: From Graphs and Back](https://mlsanity.com/paper/2210.00376/) (2022)
- [Spatiotemporal Data Mining: A Survey on Challenges and Open Problems](https://mlsanity.com/paper/2103.17128/) (2021)
- [To Spike or Not To Spike: A Digital Hardware Perspective on Deep Learning Acceleration](https://mlsanity.com/paper/2306.15749/) (2023)
- [Recent Progress in Transformer-based Medical Image Analysis](https://mlsanity.com/paper/2208.06643/) (2022)
- [CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos](https://mlsanity.com/paper/2505.18561/) (2025)
- [Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning](https://mlsanity.com/paper/2409.07238/) (2024)
- [Deep Learning for Robust Motion Segmentation with Non-Static Cameras](https://mlsanity.com/paper/2102.10929/) (2021)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [Physics-Guided Learning of Meteorological Dynamics for Weather Downscaling and Forecasting](https://mlsanity.com/paper/2505.14555/) (2025)
- [Contextual Guided Segmentation Framework for Semi-supervised Video Instance Segmentation](https://mlsanity.com/paper/2106.03330/) (2021)
- [Wide and Narrow: Video Prediction from Context and Motion](https://mlsanity.com/paper/2110.11586/) (2021)
- [FlowVOS: Weakly-Supervised Visual Warping for Detail-Preserving and Temporally Consistent Single-Shot Video Object Segmentation](https://mlsanity.com/paper/2111.10621/) (2021)
- [Pixel-Level Equalized Matching for Video Object Segmentation](https://mlsanity.com/paper/2209.03139/) (2022)
- [Learning a Fast 3D Spectral Approach to Object Segmentation and Tracking over Space and Time](https://mlsanity.com/paper/2212.08058/) (2022)
- [Automated Speaker Independent Visual Speech Recognition: A Comprehensive Survey](https://mlsanity.com/paper/2306.08314/) (2023)
- [EGR-Net: A Novel Embedding Gramian Representation CNN for Intelligent Fault Diagnosis](https://mlsanity.com/paper/2502.19199/) (2025)
- [Leveraging Motion Information for Better Self-Supervised Video Correspondence Learning](https://mlsanity.com/paper/2503.12026/) (2025)

## Abstract (arXiv)

In this paper, we present a unified, end-to-end trainable spatiotemporal CNN model for VOS, which consists of two branches, i.e., the temporal coherence branch and the spatial segmentation branch. Specifically, the temporal coherence branch pretrained in an adversarial fashion from unlabeled video data, is designed to capture the dynamic appearance and motion cues of video sequences to guide object segmentation. The spatial segmentation branch focuses on segmenting objects accurately based on the learned appearance and motion cues. To obtain accurate segmentation results, we design a coarse-to-fine process to sequentially apply a designed attention module on multi-scale feature maps, and concatenate them to produce the final prediction. In this way, the spatial segmentation branch is enforced to gradually concentrate on object regions. These two branches are jointly fine-tuned on video segmentation sequences in an end-to-end manner. Several experiments are carried out on three challenging datasets (i.e., DAVIS-2016, DAVIS-2017 and Youtube-Object) to show that our method achieves favorable performance against the state-of-the-arts. Code is available at https://github.com/longyin880815/STCNN.