# MHP-VOS: Multiple Hypotheses Propagation for Video Object Segmentation

arXiv: https://arxiv.org/abs/1904.08141 · HTML: https://mlsanity.com/paper/1904.08141/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2019.

## Built on by (most cited first, 20 of 20)

- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos](https://mlsanity.com/paper/2303.07863/) (2023)
- [Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning](https://mlsanity.com/paper/2203.14333/) (2022)
- [Region Aware Video Object Segmentation with Deep Motion Modeling](https://mlsanity.com/paper/2207.10258/) (2022)
- [Temporally Consistent Referring Video Object Segmentation with Hybrid Memory](https://mlsanity.com/paper/2403.19407/) (2024)
- [Spatiotemporal Data Mining: A Survey on Challenges and Open Problems](https://mlsanity.com/paper/2103.17128/) (2021)
- [LSVOS Challenge Report: Large-scale Complex and Long Video Object Segmentation](https://mlsanity.com/paper/2409.05847/) (2024)
- [MSN: Efficient Online Mask Selection Network for Video Instance Segmentation](https://mlsanity.com/paper/2106.10452/) (2021)
- [OneVOS: Unifying Video Object Segmentation with All-in-One Transformer Framework](https://mlsanity.com/paper/2403.08682/) (2024)
- [Efficient Long-Short Temporal Attention Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2309.11707/) (2023)
- [Point-VOS: Pointing Up Video Object Segmentation](https://mlsanity.com/paper/2402.05917/) (2024)
- [Jointly Visual- and Semantic-Aware Graph Memory Networks for Temporal Sentence Localization in Videos](https://mlsanity.com/paper/2303.01046/) (2023)
- [Transform-Equivariant Consistency Learning for Temporal Sentence Grounding](https://mlsanity.com/paper/2305.04123/) (2023)
- [Contextual Guided Segmentation Framework for Semi-supervised Video Instance Segmentation](https://mlsanity.com/paper/2106.03330/) (2021)
- [Boosting Video Object Segmentation based on Scale Inconsistency](https://mlsanity.com/paper/2205.01197/) (2022)
- [Lifelong Learning Using a Dynamically Growing Tree of Sub-networks for Domain Generalization in Video Object Segmentation](https://mlsanity.com/paper/2405.19525/) (2024)
- [Leveraging Motion Information for Better Self-Supervised Video Correspondence Learning](https://mlsanity.com/paper/2503.12026/) (2025)
- [Enhancing Self-Supervised Fine-Grained Video Object Tracking with Dynamic Memory Prediction](https://mlsanity.com/paper/2504.21692/) (2025)

## Abstract (arXiv)

We address the problem of semi-supervised video object segmentation (VOS), where the masks of objects of interests are given in the first frame of an input video. To deal with challenging cases where objects are occluded or missing, previous work relies on greedy data association strategies that make decisions for each frame individually. In this paper, we propose a novel approach to defer the decision making for a target object in each frame, until a global view can be established with the entire video being taken into consideration. Our approach is in the same spirit as Multiple Hypotheses Tracking (MHT) methods, making several critical adaptations for the VOS problem. We employ the bounding box (bbox) hypothesis for tracking tree formation, and the multiple hypotheses are spawned by propagating the preceding bbox into the detected bbox proposals within a gated region starting from the initial object mask in the first frame. The gated region is determined by a gating scheme which takes into account a more comprehensive motion model rather than the simple Kalman filtering model in traditional MHT. To further design more customized algorithms tailored for VOS, we develop a novel mask propagation score instead of the appearance similarity score that could be brittle due to large deformations. The mask propagation score, together with the motion score, determines the affinity between the hypotheses during tree pruning. Finally, a novel mask merging strategy is employed to handle mask conflicts between objects. Extensive experiments on challenging datasets demonstrate the effectiveness of the proposed method, especially in the case of object missing.