mlsanityData to 7 Oct 2026

MOSE: A New Dataset for Video Object Segmentation in Complex Scenes

arXiv 2302.01872 ↗First posted 3 Feb 2023cs.CVcited by 232

Not on the map. Why?

Abstract (arXiv)

Video object segmentation (VOS) aims at segmenting a particular object throughout the entire video clip sequence. The state-of-the-art VOS methods have achieved excellent performance (e.g., 90+% J&F) on existing datasets. However, since the target objects in these existing datasets are usually relatively salient, dominant, and isolated, VOS under complex scenes has rarely been studied. To revisit VOS and make it more applicable in the real world, we collect a new VOS dataset called coMplex video Object SEgmentation (MOSE) to study the tracking and segmenting objects in complex environments. MOSE contains 2,149 video clips and 5,200 objects from 36 categories, with 431,725 high-quality object segmentation masks. The most notable feature of MOSE dataset is complex scenes with crowded and occluded objects. The target objects in the videos are commonly occluded by others and disappear in some frames. To analyze the proposed MOSE dataset, we benchmark 18 existing VOS methods under 4 different settings on the proposed MOSE dataset and conduct comprehensive comparisons. The experiments show that current VOS algorithms cannot well perceive objects in complex scenes. For example, under the semi-supervised VOS setting, the highest J&F by existing state-of-the-art VOS methods is only 59.4% on MOSE, much lower than their ~90% J&F performance on DAVIS. The results reveal that although excellent performance has been achieved on existing benchmarks, there are unresolved challenges under complex scenes and more efforts are desired to explore these challenges in the future. The proposed MOSE dataset has been released at https://henghuiding.github.io/MOSE.

Key facts. Cited by 232 papers in the map (data through 7 Oct 2026). Builds on 89 of its 130 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

89 of 130 references.

References · 89

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  7. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  8. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  9. 2019Video Instance Segmentationcited by 391
  10. 2018Fast Online Object Tracking and Segmentation: A Unifying Approachcited by 272
  11. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  12. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
38 more
  1. 2016One-Shot Video Object Segmentationcited by 233
  2. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 228
  3. 2021Associating Objects with Transformers for Video Object Segmentationcited by 225
  4. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 200
  5. 2019FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentationcited by 178
  6. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  7. 2016Learning Video Object Segmentation from Static Imagescited by 151
  8. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 146
  9. 2020Collaborative Video Object Segmentation by Foreground-Background Integrationcited by 142
  10. 2020Video Panoptic Segmentationcited by 141
  11. 2020See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networkscited by 138
  12. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  13. 2021A Survey on Deep Learning Technique for Video Segmentationcited by 132
  14. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  15. 2020STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videoscited by 121
  16. 2020Video Object Segmentation with Episodic Graph Memory Networkscited by 118
  17. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  18. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  19. 2019Boundary-Aware Feature Propagation for Scene Segmentationcited by 107
  20. 2021SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentationcited by 106
  21. 2021Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayerscited by 104
  22. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  23. 2020Self-Supervised Scene De-occlusioncited by 96
  24. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 95
  25. 2020ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentationcited by 95
  26. 2017SegFlow: Joint Learning for Video Object Segmentation and Optical Flowcited by 92
  27. 2019RVOS: End-to-End Recurrent Network for Video Object Segmentationcited by 88
  28. 2020Zero-Shot Video Object Segmentation via Attentive Graph Neural Networkscited by 84
  29. 2017FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videoscited by 77
  30. 2019Online Multi-Object Tracking with Dual Matching Attention Networkscited by 77
  31. 2018Blazingly Fast Video Object Segmentation with Pixel-Wise Metric Learningcited by 74
  32. 2019Spatial-Temporal Relation Networks for Multi-Object Trackingcited by 74
  33. 2019Motion Guided Attention for Video Salient Object Detectioncited by 72
  34. 2021Recovering the Unbiased Scene Graphs from the Biased Onescited by 71
  35. 2017Learning Video Object Segmentation with Visual Memorycited by 66
  36. 2021Learning Position and Target Consistency for Memory-based Video Object Segmentationcited by 63
  37. 2017Repulsion Loss: Detecting Pedestrians in a Crowdcited by 62
  38. 2019Semantic Correlation Promoted Shape-Variant Context for Segmentationcited by 61

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 60; 2025: 83; 2026: 59.

232 papers.

Sort:

By area:

  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  2. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  3. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  4. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  5. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  6. 2023SegGPT: Segmenting Everything In Contextcited by 193
  7. 2023Putting the Object Back into Video Object Segmentationcited by 182
  8. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  9. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  10. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  11. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  12. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
38 more
  1. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 83
  2. 2023Segment Anything Meets Point Trackingcited by 81
  3. 2024SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Treecited by 77
  4. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  5. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  6. 2024OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuningcited by 64
  7. 2024SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentationcited by 63
  8. 2024A Distractor-Aware Memory for Visual Object Tracking with SAM2cited by 63
  9. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  10. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 59
  11. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  12. 2024ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Modelscited by 54
  13. 2023XMem++: Production-level Video Segmentation From Few Annotated Framescited by 53
  14. 2025VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion Controlcited by 42
  15. 2024LVOS: A Benchmark for Large-scale Long-term Video Object Segmentationcited by 41
  16. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentationcited by 40
  17. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  18. 2023GREC: Generalized Referring Expression Comprehensioncited by 39
  19. 2023Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentationcited by 36
  20. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  21. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  22. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 32
  23. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  24. 2023SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Processcited by 28
  25. 2024UniVS: Unified and Universal Video Segmentation with Prompts as Queriescited by 26
  26. 2025TransAnimate: Taming Layer Diffusion to Generate RGBA Videocited by 24
  27. 2025Advancing Complex Video Object Segmentation via Progressive Concept Constructioncited by 24
  28. 2023Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentationcited by 23
  29. 2023UniRef++: Segment Every Reference Object in Spatial and Temporal Spacescited by 22
  30. 2024Object Segmentation-Assisted Inter Prediction for Versatile Video Codingcited by 22
  31. 2024RMem: Restricted Memory Banks Improve Video Object Segmentationcited by 22
  32. 2025Multimodal Referring Segmentation: A Surveycited by 22
  33. 2025EdgeTAM: On-Device Track Anything Modelcited by 21
  34. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  35. 2024Efficient Track Anythingcited by 20
  36. 2025ReferSplat: Referring Segmentation in 3D Gaussian Splattingcited by 20
  37. 2023Diversifying Spatial-Temporal Perception for Video Domain Generalizationcited by 19
  38. 2023Learning the What and How of Annotation in Video Object Segmentationcited by 19

50 most cited of 232 All 232 (JSON)