# Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation

arXiv: https://arxiv.org/abs/2206.03789 · HTML: https://mlsanity.com/paper/2206.03789/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [Referring Image Segmentation via Cross-Modal Progressive Comprehension](https://mlsanity.com/paper/2010.00514/) (2020)
- [Actor and Action Video Segmentation from a Sentence](https://mlsanity.com/paper/1803.07485/) (2018)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [Linguistic Structure Guided Context Modeling for Referring Image Segmentation](https://mlsanity.com/paper/2010.00515/) (2020)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [Locate then Segment: A Strong Pipeline for Referring Image Segmentation](https://mlsanity.com/paper/2103.16284/) (2021)
- [Motion Guided Attention for Video Salient Object Detection](https://mlsanity.com/paper/1909.07061/) (2019)
- [SwiftNet: Real-time Video Object Segmentation](https://mlsanity.com/paper/2102.04604/) (2021)
- [Semi-Supervised Video Salient Object Detection Using Pseudo-Labels](https://mlsanity.com/paper/1908.04051/) (2019)
- [Multi-Fiber Networks for Video Recognition](https://mlsanity.com/paper/1807.11195/) (2018)
- [Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation](https://mlsanity.com/paper/2105.06818/) (2021)
- [Learning Dynamic Network Using a Reuse Gate Function in Semi-supervised Video Object Segmentation](https://mlsanity.com/paper/2012.11655/) (2020)
- [Referring Segmentation in Images and Videos with Cross-Modal Self-Attention Network](https://mlsanity.com/paper/2102.04762/) (2021)
- [M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers](https://mlsanity.com/paper/2104.01122/) (2021)

## Built on by (most cited first, 30 of 50)

- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [VISA: Reasoning Video Object Segmentation via Large Language Models](https://mlsanity.com/paper/2407.11325/) (2024)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos](https://mlsanity.com/paper/2409.19603/) (2024)
- [Referring Multi-Object Tracking](https://mlsanity.com/paper/2303.03366/) (2023)
- [OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation](https://mlsanity.com/paper/2307.09356/) (2023)
- [SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation](https://mlsanity.com/paper/2305.17011/) (2023)
- [Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation](https://mlsanity.com/paper/2404.03645/) (2024)
- [Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation](https://mlsanity.com/paper/2305.16318/) (2023)
- [Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensus](https://mlsanity.com/paper/2207.01203/) (2022)
- [Spectrum-guided Multi-granularity Referring Video Object Segmentation](https://mlsanity.com/paper/2307.13537/) (2023)
- [LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation](https://mlsanity.com/paper/2404.19326/) (2024)
- [SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation](https://mlsanity.com/paper/2411.17646/) (2024)
- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [Tracking with Human-Intent Reasoning](https://mlsanity.com/paper/2312.17448/) (2023)
- [Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation](https://mlsanity.com/paper/2309.09501/) (2023)
- [Universal Segmentation at Arbitrary Granularity with Language Instruction](https://mlsanity.com/paper/2312.01623/) (2023)
- [Temporal Collection and Distribution for Referring Video Object Segmentation](https://mlsanity.com/paper/2309.03473/) (2023)
- [Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation](https://mlsanity.com/paper/2403.12042/) (2024)
- [RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation](https://mlsanity.com/paper/2307.00997/) (2023)
- [UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces](https://mlsanity.com/paper/2312.15715/) (2023)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Temporally Consistent Referring Video Object Segmentation with Hybrid Memory](https://mlsanity.com/paper/2403.19407/) (2024)
- [PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding](https://mlsanity.com/paper/2208.05647/) (2022)
- [Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation](https://mlsanity.com/paper/2309.13237/) (2023)
- [CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos](https://mlsanity.com/paper/2505.18561/) (2025)
- [PaintSeg: Training-free Segmentation via Painting](https://mlsanity.com/paper/2305.19406/) (2023)
- [Vision-Aware Text Features in Referring Image Segmentation: From Object Understanding to Context Understanding](https://mlsanity.com/paper/2404.08590/) (2024)
- [MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation](https://mlsanity.com/paper/2501.13667/) (2025)

## Abstract (arXiv)

Referring video object segmentation aims to predict foreground labels for objects referred by natural language expressions in videos. Previous methods either depend on 3D ConvNets or incorporate additional 2D ConvNets as encoders to extract mixed spatial-temporal features. However, these methods suffer from spatial misalignment or false distractors due to delayed and implicit spatial-temporal interaction occurring in the decoding phase. To tackle these limitations, we propose a Language-Bridged Duplex Transfer (LBDT) module which utilizes language as an intermediary bridge to accomplish explicit and adaptive spatial-temporal interaction earlier in the encoding phase. Concretely, cross-modal attention is performed among the temporal encoder, referring words and the spatial encoder to aggregate and transfer language-relevant motion and appearance information. In addition, we also propose a Bilateral Channel Activation (BCA) module in the decoding phase for further denoising and highlighting the spatial-temporal consistent features via channel-wise activation. Extensive experiments show our method achieves new state-of-the-art performances on four popular benchmarks with 6.8% and 6.9% absolute AP gains on A2D Sentences and J-HMDB Sentences respectively, while consuming around 7x less computational overhead.