# CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing

arXiv: https://arxiv.org/abs/1910.00132 · HTML: https://mlsanity.com/paper/1910.00132/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 29 of 29)

- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [ControlNeXt: Powerful and Efficient Control for Image and Video Generation](https://mlsanity.com/paper/2408.06070/) (2024)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [Hierarchical Memory Matching Network for Video Object Segmentation](https://mlsanity.com/paper/2109.11404/) (2021)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [XMem++: Production-level Video Segmentation From Few Annotated Frames](https://mlsanity.com/paper/2307.15958/) (2023)
- [Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules](https://mlsanity.com/paper/2105.04836/) (2021)
- [Region Aware Video Object Segmentation with Deep Motion Modeling](https://mlsanity.com/paper/2207.10258/) (2022)
- [Tackling Background Distraction in Video Object Segmentation](https://mlsanity.com/paper/2207.06953/) (2022)
- [Temporally Consistent Referring Video Object Segmentation with Hybrid Memory](https://mlsanity.com/paper/2403.19407/) (2024)
- [Pixel-Level Bijective Matching for Video Object Segmentation](https://mlsanity.com/paper/2110.01644/) (2021)
- [Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2](https://mlsanity.com/paper/2510.24195/) (2025)
- [Learning with Capsules: A Survey](https://mlsanity.com/paper/2206.02664/) (2022)
- [CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos](https://mlsanity.com/paper/2505.18561/) (2025)
- [Weakly Supervised Grounding for VQA in Vision-Language Transformers](https://mlsanity.com/paper/2207.02334/) (2022)
- [Latest Object Memory Management for Temporally Consistent Video Instance Segmentation](https://mlsanity.com/paper/2507.19754/) (2025)
- [Deformable Capsules for Object Detection](https://mlsanity.com/paper/2104.05031/) (2021)
- [Efficient Video Object Segmentation via Modulated Cross-Attention Memory](https://mlsanity.com/paper/2403.17937/) (2024)
- [Segment Anything Across Shots: A Method and Benchmark](https://mlsanity.com/paper/2511.13715/) (2025)
- [DPR-CAE: Capsule Autoencoder with Dynamic Part Representation for Image Parsing](https://mlsanity.com/paper/2104.14735/) (2021)
- [Self-Supervised Video Object Segmentation via Cutout Prediction and Tagging](https://mlsanity.com/paper/2204.10846/) (2022)
- [Pixel-Level Equalized Matching for Video Object Segmentation](https://mlsanity.com/paper/2209.03139/) (2022)
- [Instance-Agnostic Geometry and Contact Dynamics Learning](https://mlsanity.com/paper/2309.05832/) (2023)
- [Leveraging Motion Information for Better Self-Supervised Video Correspondence Learning](https://mlsanity.com/paper/2503.12026/) (2025)
- [EquiCaps: Predictor-Free Pose-Aware Pre-Trained Capsule Networks](https://mlsanity.com/paper/2506.09895/) (2025)

## Abstract (arXiv)

In this work we propose a capsule-based approach for semi-supervised video object segmentation. Current video object segmentation methods are frame-based and often require optical flow to capture temporal consistency across frames which can be difficult to compute. To this end, we propose a video based capsule network, CapsuleVOS, which can segment several frames at once conditioned on a reference frame and segmentation mask. This conditioning is performed through a novel routing algorithm for attention-based efficient capsule selection. We address two challenging issues in video object segmentation: 1) segmentation of small objects and 2) occlusion of objects across time. The issue of segmenting small objects is addressed with a zooming module which allows the network to process small spatial regions of the video. Apart from this, the framework utilizes a novel memory module based on recurrent networks which helps in tracking objects when they move out of frame or are occluded. The network is trained end-to-end and we demonstrate its effectiveness on two benchmark video object segmentation datasets; it outperforms current offline approaches on the Youtube-VOS dataset while having a run-time that is almost twice as fast as competing methods. The code is publicly available at https://github.com/KevinDuarte/CapsuleVOS.