# Advancing Complex Video Object Segmentation via Progressive Concept Construction

arXiv: https://arxiv.org/abs/2507.15852 · HTML: https://mlsanity.com/paper/2507.15852/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 71 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Builds on (most cited first, 30 of 57)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://mlsanity.com/paper/2407.03320/) (2024)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [VISA: Reasoning Video Object Segmentation via Large Language Models](https://mlsanity.com/paper/2407.11325/) (2024)
- [Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction](https://mlsanity.com/paper/2501.03218/) (2025)
- [Kernelized Memory Network for Video Object Segmentation](https://mlsanity.com/paper/2007.08270/) (2020)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement](https://mlsanity.com/paper/2010.07958/) (2020)
- [One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos](https://mlsanity.com/paper/2409.19603/) (2024)

## Built on by (most cited first, 26 of 26)

- [SIM-CoT: Supervised Implicit Chain-of-Thought](https://mlsanity.com/paper/2509.20317/) (2025)
- [Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2510.27606/) (2025)
- [ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning](https://mlsanity.com/paper/2512.05111/) (2025)
- [STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence](https://mlsanity.com/paper/2510.24693/) (2025)
- [Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation](https://mlsanity.com/paper/2602.03595/) (2026)
- [UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation](https://mlsanity.com/paper/2512.07831/) (2025)
- [Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds](https://mlsanity.com/paper/2608.02636/) (2026)
- [Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search](https://mlsanity.com/paper/2602.04454/) (2026)
- [Visual Self-Refine: A Pixel-Guided Paradigm for Accurate Chart Parsing](https://mlsanity.com/paper/2602.16455/) (2026)
- [LSVOS 2025 Challenge Report: Recent Advances in Complex Video Object Segmentation](https://mlsanity.com/paper/2510.11063/) (2025)
- [TripVVT: A Large-Scale Triplet Dataset and a Coarse-Mask Baseline for In-the-Wild Video Virtual Try-On](https://mlsanity.com/paper/2604.27958/) (2026)
- [SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction](https://mlsanity.com/paper/2605.20110/) (2026)
- [Segment Anything with Motion, Geometry, and Semantic Adaptation for Complex Nonlinear Visual Object Tracking](https://mlsanity.com/paper/2605.22538/) (2026)
- [Pseudo-Label Enhanced Cascaded Framework: 2nd Technical Report for LSVOS 2025 VOS Track](https://mlsanity.com/paper/2509.14901/) (2025)
- [SAMSON: 3rd Place Solution of LSVOS 2025 VOS Challenge](https://mlsanity.com/paper/2509.17500/) (2025)
- [The 1st Solution for MOSEv2 Challenge 2025: Long-term and Concept-aware Video Segmentation via SeC](https://mlsanity.com/paper/2509.19183/) (2025)
- [2nd Place Report of MOSEv2 Challenge 2025: Concept Guided Video Object Segmentation via SeC](https://mlsanity.com/paper/2509.23838/) (2025)
- [Re-Prompting SAM 3 via Object Retrieval: 3rd of the 5th PVUW MOSE Track](https://mlsanity.com/paper/2603.23788/) (2026)
- [Advancing Complex Video Object Segmentation via Tracking-Enhanced Prompt: The 1st Winner for 5th PVUW MOSE Challenge](https://mlsanity.com/paper/2604.00395/) (2026)
- [Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games](https://mlsanity.com/paper/2606.19338/) (2026)
- [Gaze Target Estimation Anywhere with Concepts](https://mlsanity.com/paper/2608.11367/) (2026)
- [VOS-Agent: The 1st Place Solution for the 8th LSVOS Challenge (MOSEv2 Track)](https://mlsanity.com/paper/2608.12721/) (2026)
- [Competitive Memory Readout for Robust Video Object Segmentation: 2nd Place Technical Report for the MOSEv2 Track of the 8th LSVOS Challenge](https://mlsanity.com/paper/2608.22064/) (2026)
- [ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation](https://mlsanity.com/paper/2609.03756/) (2026)
- [When Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation](https://mlsanity.com/paper/2610.02946/) (2026)
- [When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation](https://mlsanity.com/paper/2610.15929/) (2026)

## Abstract (arXiv)

We propose Segment Concept (SeC), a concept-driven video object segmentation (VOS) framework that shifts from conventional feature matching to the progressive construction and utilization of high-level, object-centric representations. SeC employs Large Vision-Language Models (LVLMs) to integrate visual cues across diverse frames, constructing robust conceptual priors. To balance semantic reasoning with computational overhead, SeC forwards the LVLMs only when a new scene appears, injecting concept-level features at those points. To rigorously assess VOS methods in scenarios demanding high-level conceptual reasoning and robust semantic understanding, we introduce the Semantic Complex Scenarios Video Object Segmentation benchmark (SeCVOS). SeCVOS comprises 160 manually annotated multi-scenario videos designed to challenge models with substantial appearance variations and dynamic scene transformations. Empirical evaluations demonstrate that SeC substantially outperforms state-of-the-art approaches, including SAM 2 and its advanced variants, on both SeCVOS and standard VOS benchmarks. In particular, SeC achieves an 11.8-point improvement over SAM 2.1 on SeCVOS, establishing a new state-of-the-art in concept-aware VOS.