# MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes

arXiv: https://arxiv.org/abs/2508.05630 · HTML: https://mlsanity.com/paper/2508.05630/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 132 of its 186 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 30 of 132)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)

## Built on by (most cited first, 30 of 62)

- [Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding](https://mlsanity.com/paper/2601.10611/) (2026)
- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing](https://mlsanity.com/paper/2603.19224/) (2026)
- [PISCO: Precise Video Instance Insertion with Sparse Control](https://mlsanity.com/paper/2602.08277/) (2026)
- [The 1st Solution for 7th LSVOS RVOS Track: SaSaSa2VA](https://mlsanity.com/paper/2509.16972/) (2025)
- [GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation](https://mlsanity.com/paper/2601.05244/) (2026)
- [LSVOS 2025 Challenge Report: Recent Advances in Complex Video Object Segmentation](https://mlsanity.com/paper/2510.11063/) (2025)
- [AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios](https://mlsanity.com/paper/2511.21053/) (2025)
- [3AM: 3egment Anything with Geometric Consistency in Videos](https://mlsanity.com/paper/2601.08831/) (2026)
- [Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Engine](https://mlsanity.com/paper/2511.13713/) (2025)
- [Segment Anything Across Shots: A Method and Benchmark](https://mlsanity.com/paper/2511.13715/) (2025)
- [Geometry-Guided Camera Motion Understanding in VideoLLMs](https://mlsanity.com/paper/2603.13119/) (2026)
- [CRISP: Contrastive Residual Injection and Semantic Prompting for Continual Video Instance Segmentation](https://mlsanity.com/paper/2508.10432/) (2025)
- [Ref-SAM3D: Bridging SAM3D with Text for Reference 3D Reconstruction](https://mlsanity.com/paper/2511.19426/) (2025)
- [SAM3-DMS: Decoupled Memory Selection for Multi-target Video Segmentation of SAM3](https://mlsanity.com/paper/2601.09699/) (2026)
- [A Hyperbolic Perspective on Hierarchical Structure in Object-Centric Scene Representations](https://mlsanity.com/paper/2603.14022/) (2026)
- [3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio](https://mlsanity.com/paper/2603.23126/) (2026)
- [Live Interactive Training for Video Segmentation](https://mlsanity.com/paper/2603.26929/) (2026)
- [The 1st Winner for 5th PVUW MeViS-Text Challenge: Strong MLLMs Meet SAM3 for Referring Video Object Segmentation](https://mlsanity.com/paper/2604.00404/) (2026)
- [2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA](https://mlsanity.com/paper/2604.23935/) (2026)
- [Report of the 5th PVUW Challenge: Towards More Diverse Modalities in Pixel-Level Understanding](https://mlsanity.com/paper/2604.26031/) (2026)
- [Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2605.08874/) (2026)
- [Pseudo-Label Enhanced Cascaded Framework: 2nd Technical Report for LSVOS 2025 VOS Track](https://mlsanity.com/paper/2509.14901/) (2025)
- [Enriched Feature Representation and Motion Prediction Module for MOSEv2 Track of 7th LSVOS Challenge: 3rd Place Solution](https://mlsanity.com/paper/2509.15781/) (2025)
- [SAMSON: 3rd Place Solution of LSVOS 2025 VOS Challenge](https://mlsanity.com/paper/2509.17500/) (2025)
- [The 1st Solution for MOSEv2 Challenge 2025: Long-term and Concept-aware Video Segmentation via SeC](https://mlsanity.com/paper/2509.19183/) (2025)
- [2nd Place Report of MOSEv2 Challenge 2025: Concept Guided Video Object Segmentation via SeC](https://mlsanity.com/paper/2509.23838/) (2025)
- [The 1st Solution for MOSEv1 Challenge on LSVOS 2025: CGFSeg](https://mlsanity.com/paper/2509.25738/) (2025)
- [Benchmarking SAM2-based Trackers on FMOX](https://mlsanity.com/paper/2512.09633/) (2025)

## Abstract (arXiv)

Video object segmentation (VOS) aims to segment specified target objects throughout a video. Although state-of-the-art methods have achieved impressive performance (e.g., 90+% J&F) on benchmarks such as DAVIS and YouTube-VOS, these datasets primarily contain salient, dominant, and isolated objects, limiting their generalization to real-world scenarios. To bridge this gap, the coMplex video Object SEgmentation (MOSEv1) dataset was introduced to facilitate VOS research in complex scenes. Building on the foundations and insights of MOSEv1, we present MOSEv2, a significantly more challenging dataset designed to further advance VOS methods under real-world conditions. MOSEv2 consists of 5,024 videos and 701,976 high-quality masks for 10,074 objects across 200 categories. Compared to its predecessor, MOSEv2 introduces much greater scene complexity, including {more frequent object disappearance and reappearance, severe occlusions and crowding, smaller objects, as well as a range of new challenges such as adverse weather (e.g., rain, snow, fog), low-light scenes (e.g., nighttime, underwater), multi-shot sequences, camouflaged objects, non-physical targets (e.g., shadows, reflections), and scenarios requiring external knowledge.} We benchmark 20 representative VOS methods under 5 different settings and observe consistent performance drops on MOSEv2. For example, SAM2 drops from 76.4% on MOSEv1 to only 50.9% on MOSEv2. We further evaluate 9 video object tracking methods and observe similar declines, demonstrating that MOSEv2 poses challenges across tasks. These results highlight that despite strong performance on existing datasets, current VOS methods still fall short under real-world complexities. Based on our analysis of the observed challenges, we further propose several practical tricks that enhance model performance. MOSEv2 is publicly available at https://MOSE.video.