# Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation

arXiv: https://arxiv.org/abs/2404.03645 · HTML: https://mlsanity.com/paper/2404.03645/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 65 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Generalized Decoding for Pixel, Image, and Language](https://mlsanity.com/paper/2212.11270/) (2022)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Detection and Tracking Meet Drones Challenge](https://mlsanity.com/paper/2001.06303/) (2020)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](https://mlsanity.com/paper/2302.07387/) (2023)
- [End-to-End Referring Video Object Segmentation with Multimodal Transformers](https://mlsanity.com/paper/2111.14821/) (2021)
- [Recurrent Multimodal Interaction for Referring Image Segmentation](https://mlsanity.com/paper/1703.07939/) (2017)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [ReSTR: Convolution-free Referring Image Segmentation Using Transformers](https://mlsanity.com/paper/2203.16768/) (2022)

## Built on by (most cited first, 30 of 55)

- [One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos](https://mlsanity.com/paper/2409.19603/) (2024)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation](https://mlsanity.com/paper/2411.17646/) (2024)
- [SegPoint: Segment Any Point Cloud via Large Language Model](https://mlsanity.com/paper/2407.13761/) (2024)
- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations](https://mlsanity.com/paper/2501.14607/) (2025)
- [The Devil is in Temporal Token: High Quality Video Reasoning Segmentation](https://mlsanity.com/paper/2501.08549/) (2025)
- [RefMask3D: Language-Guided Transformer for 3D Referring Segmentation](https://mlsanity.com/paper/2407.18244/) (2024)
- [Bootstrapping Referring Multi-Object Tracking](https://mlsanity.com/paper/2406.05039/) (2024)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level](https://mlsanity.com/paper/2411.09921/) (2024)
- [ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation](https://mlsanity.com/paper/2412.09754/) (2024)
- [PVUW 2024 Challenge on Complex Video Understanding: Methods and Results](https://mlsanity.com/paper/2406.17005/) (2024)
- [Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation](https://mlsanity.com/paper/2507.22886/) (2025)
- [ReferEverything: Towards Segmenting Everything We Can Speak of in Videos](https://mlsanity.com/paper/2410.23287/) (2024)
- [Deforming Videos to Masks: Flow Matching for Referring Video Segmentation](https://mlsanity.com/paper/2510.06139/) (2025)
- [Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2405.15169/) (2024)
- [LSVOS Challenge Report: Large-scale Complex and Long Video Object Segmentation](https://mlsanity.com/paper/2409.05847/) (2024)
- [MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation](https://mlsanity.com/paper/2501.13667/) (2025)
- [MOVE: Motion-Guided Few-Shot Video Object Segmentation](https://mlsanity.com/paper/2507.22061/) (2025)
- [VideoSeg-R1:Reasoning Video Object Segmentation via Reinforcement Learning](https://mlsanity.com/paper/2511.16077/) (2025)
- [PVUW 2025 Challenge Report: Advances in Pixel-level Understanding of Complex Videos in the Wild](https://mlsanity.com/paper/2504.11326/) (2025)
- [Implicit Counterfactual Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2507.20740/) (2025)
- [MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding](https://mlsanity.com/paper/2510.09274/) (2025)
- [Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding](https://mlsanity.com/paper/2510.20244/) (2025)
- [AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation](https://mlsanity.com/paper/2603.23489/) (2026)
- [The 2nd Solution for LSVOS Challenge RVOS Track: Spatial-temporal Refinement for Consistent Semantic Segmentation](https://mlsanity.com/paper/2408.12447/) (2024)
- [Referring Video Object Segmentation via Language-aligned Track Selection](https://mlsanity.com/paper/2412.01136/) (2024)
- [Find First, Track Next: Decoupling Identification and Propagation in Referring Video Object Segmentation](https://mlsanity.com/paper/2503.03492/) (2025)
- [The 1st Solution for 4th PVUW MeViS Challenge: Unleashing the Potential of Large Multimodal Models for Referring Video Segmentation](https://mlsanity.com/paper/2504.05178/) (2025)

## Abstract (arXiv)

Referring video segmentation relies on natural language expressions to identify and segment objects, often emphasizing motion clues. Previous works treat a sentence as a whole and directly perform identification at the video-level, mixing up static image-level cues with temporal motion cues. However, image-level features cannot well comprehend motion cues in sentences, and static cues are not crucial for temporal perception. In fact, static cues can sometimes interfere with temporal perception by overshadowing motion cues. In this work, we propose to decouple video-level referring expression understanding into static and motion perception, with a specific emphasis on enhancing temporal comprehension. Firstly, we introduce an expression-decoupling module to make static cues and motion cues perform their distinct role, alleviating the issue of sentence embeddings overlooking motion cues. Secondly, we propose a hierarchical motion perception module to capture temporal information effectively across varying timescales. Furthermore, we employ contrastive learning to distinguish the motions of visually similar objects. These contributions yield state-of-the-art performance across five datasets, including a remarkable 9.2% J&F improvement on the challenging MeViS dataset. Code is available at https://github.com/heshuting555/DsHmp.