# SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation

arXiv: https://arxiv.org/abs/2305.17011 · HTML: https://mlsanity.com/paper/2305.17011/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](https://mlsanity.com/paper/2112.14757/) (2021)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](https://mlsanity.com/paper/2302.07387/) (2023)
- [End-to-End Referring Video Object Segmentation with Multimodal Transformers](https://mlsanity.com/paper/2111.14821/) (2021)
- [Referring Image Segmentation via Cross-Modal Progressive Comprehension](https://mlsanity.com/paper/2010.00514/) (2020)
- [VLT: Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2210.15871/) (2022)
- [Actor and Action Video Segmentation from a Sentence](https://mlsanity.com/paper/1803.07485/) (2018)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [VITA: Video Instance Segmentation via Object Token Association](https://mlsanity.com/paper/2206.04403/) (2022)
- [Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping](https://mlsanity.com/paper/2305.11003/) (2023)
- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [Semantic Correlation Promoted Shape-Variant Context for Segmentation](https://mlsanity.com/paper/1909.02651/) (2019)
- [Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation](https://mlsanity.com/paper/2206.03789/) (2022)
- [Rethinking Cross-modal Interaction from a Top-down Perspective for Referring Video Object Segmentation](https://mlsanity.com/paper/2106.01061/) (2021)
- [Global Knowledge Calibration for Fast Open-Vocabulary Segmentation](https://mlsanity.com/paper/2303.09181/) (2023)

## Built on by (most cited first, 30 of 57)

- [One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos](https://mlsanity.com/paper/2409.19603/) (2024)
- [ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models](https://mlsanity.com/paper/2412.00447/) (2024)
- [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025)
- [General Object Foundation Model for Images and Videos at Scale](https://mlsanity.com/paper/2312.09158/) (2023)
- [Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation](https://mlsanity.com/paper/2404.03645/) (2024)
- [Open-Vocabulary Segmentation with Semantic-Assisted Calibration](https://mlsanity.com/paper/2312.04089/) (2023)
- [SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation](https://mlsanity.com/paper/2411.17646/) (2024)
- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning](https://mlsanity.com/paper/2505.22596/) (2025)
- [ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations](https://mlsanity.com/paper/2501.14607/) (2025)
- [Universal Segmentation at Arbitrary Granularity with Language Instruction](https://mlsanity.com/paper/2312.01623/) (2023)
- [RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation](https://mlsanity.com/paper/2307.00997/) (2023)
- [LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation](https://mlsanity.com/paper/2306.08736/) (2023)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes](https://mlsanity.com/paper/2506.01558/) (2025)
- [Temporally Consistent Referring Video Object Segmentation with Hybrid Memory](https://mlsanity.com/paper/2403.19407/) (2024)
- [Deforming Videos to Masks: Flow Matching for Referring Video Segmentation](https://mlsanity.com/paper/2510.06139/) (2025)
- [Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics](https://mlsanity.com/paper/2503.12840/) (2025)
- [CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos](https://mlsanity.com/paper/2505.18561/) (2025)
- [Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation](https://mlsanity.com/paper/2602.03595/) (2026)
- [Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification](https://mlsanity.com/paper/2403.08271/) (2024)
- [MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation](https://mlsanity.com/paper/2501.13667/) (2025)
- [DreamLight: Towards Harmonious and Consistent Image Relighting](https://mlsanity.com/paper/2506.14549/) (2025)
- [Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2506.16058/) (2025)
- [Complementary and Contrastive Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2510.10051/) (2025)
- [The 1st Solution for 4th PVUW MeViS Challenge: Unleashing the Potential of Large Multimodal Models for Referring Video Segmentation](https://mlsanity.com/paper/2504.05178/) (2025)
- [MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding](https://mlsanity.com/paper/2510.09274/) (2025)
- [UNINEXT-Cutie: The 1st Solution for LSVOS Challenge RVOS Track](https://mlsanity.com/paper/2408.10129/) (2024)
- [Find First, Track Next: Decoupling Identification and Propagation in Referring Video Object Segmentation](https://mlsanity.com/paper/2503.03492/) (2025)
- [VoCap: Video Object Captioning and Segmentation from Any Prompt](https://mlsanity.com/paper/2508.21809/) (2025)

## Abstract (arXiv)

This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. Recent approaches model the RVOS task as a sequence prediction problem and perform multi-modal interaction as well as segmentation for each frame separately. However, the lack of a global view of video content leads to difficulties in effectively utilizing inter-frame relationships and understanding textual descriptions of object temporal variations. To address this issue, we propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. By associating a group of frame-level object embeddings with language tokens, SOC facilitates joint space learning across modalities and time steps. Moreover, we present multi-modal contrastive supervision to help construct well-aligned joint space at the video level. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin. Besides, the emphasis on temporal coherence enhances the segmentation stability and adaptability of our method in processing text expressions with temporal variations. Code will be available.