# MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning

arXiv: https://arxiv.org/abs/2508.04549 · HTML: https://mlsanity.com/paper/2508.04549/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 26 of 26)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions](https://mlsanity.com/paper/2407.06358/) (2024)
- [Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content](https://mlsanity.com/paper/2410.08260/) (2024)
- [Actor and Action Video Segmentation from a Sentence](https://mlsanity.com/paper/1803.07485/) (2018)
- [VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation](https://mlsanity.com/paper/2106.04632/) (2021)
- [Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation](https://mlsanity.com/paper/2404.03645/) (2024)
- [MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation](https://mlsanity.com/paper/2411.15262/) (2024)
- [Diving into Underwater: Segment Anything Model Guided Underwater Salient Instance Segmentation and A Large-scale Dataset](https://mlsanity.com/paper/2406.06039/) (2024)
- [A Bilingual, OpenWorld Video Text Dataset and End-to-end Video Text Spotter with Transformer](https://mlsanity.com/paper/2112.04888/) (2021)
- [HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation](https://mlsanity.com/paper/2503.23715/) (2025)
- [ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation](https://mlsanity.com/paper/2412.09754/) (2024)
- [BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimation](https://mlsanity.com/paper/2503.20781/) (2025)
- [MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alps](https://mlsanity.com/paper/2503.18223/) (2025)
- [Marine Video Kit: A New Marine Video Dataset for Content-based Analysis and Retrieval](https://mlsanity.com/paper/2209.11518/) (2022)
- [Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images](https://mlsanity.com/paper/2407.08027/) (2024)
- [CoralSCOP-LAT: Labeling and Analyzing Tool for Coral Reef Images with Dense Mask](https://mlsanity.com/paper/2410.20436/) (2024)
- [Results of the 2024 Video Browser Showdown](https://mlsanity.com/paper/2502.15683/) (2024)

## Abstract (arXiv)

Marine videos present significant challenges for video understanding due to the dynamics of marine objects and the surrounding environment, camera motion, and the complexity of underwater scenes. Existing video captioning datasets, typically focused on generic or human-centric domains, often fail to generalize to the complexities of the marine environment and gain insights about marine life. To address these limitations, we propose a two-stage marine object-oriented video captioning pipeline. We introduce a comprehensive video understanding benchmark that leverages the triplets of video, text, and segmentation masks to facilitate visual grounding and captioning, leading to improved marine video understanding and analysis, and marine video generation. Additionally, we highlight the effectiveness of video splitting in order to detect salient object transitions in scene changes, which significantly enrich the semantics of captioning content. Our dataset and code have been released at https://msc.hkustvgd.com.