# Multi-Attention Network for Compressed Video Referring Object Segmentation

arXiv: https://arxiv.org/abs/2207.12622 · HTML: https://mlsanity.com/paper/2207.12622/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 61 references that are in the map.
- First posted on arXiv 26 Jul 2022.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [End-to-End Referring Video Object Segmentation with Multimodal Transformers](https://mlsanity.com/paper/2111.14821/) (2021)
- [Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering](https://mlsanity.com/paper/1812.05252/) (2018)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [Actor and Action Video Segmentation from a Sentence](https://mlsanity.com/paper/1803.07485/) (2018)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [Multimodal Transformer with Multi-View Visual Representation for Image Captioning](https://mlsanity.com/paper/1905.07841/) (2019)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [Fine-grained Iterative Attention Network for TemporalLanguage Localization in Videos](https://mlsanity.com/paper/2008.02448/) (2020)
- [DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition](https://mlsanity.com/paper/1901.03460/) (2019)

## Built on by (most cited first, 28 of 28)

- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation](https://mlsanity.com/paper/2307.09356/) (2023)
- [Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation](https://mlsanity.com/paper/2404.03645/) (2024)
- [Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation](https://mlsanity.com/paper/2305.16318/) (2023)
- [Spectrum-guided Multi-granularity Referring Video Object Segmentation](https://mlsanity.com/paper/2307.13537/) (2023)
- [Annotation-free Audio-Visual Segmentation](https://mlsanity.com/paper/2305.11019/) (2023)
- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [Learning to Dub Movies via Hierarchical Prosody Models](https://mlsanity.com/paper/2212.04054/) (2022)
- [Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation](https://mlsanity.com/paper/2403.12042/) (2024)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition](https://mlsanity.com/paper/2310.00132/) (2023)
- [Towards Efficient Partially Relevant Video Retrieval with Active Moment Discovering](https://mlsanity.com/paper/2504.10920/) (2025)
- [Temporally Consistent Referring Video Object Segmentation with Hybrid Memory](https://mlsanity.com/paper/2403.19407/) (2024)
- [Dual-path Collaborative Generation Network for Emotional Video Captioning](https://mlsanity.com/paper/2408.03006/) (2024)
- [PaintSeg: Training-free Segmentation via Painting](https://mlsanity.com/paper/2305.19406/) (2023)
- [Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2606.08566/) (2026)
- [Learning More May Not Be Better: Knowledge Transferability in Vision and Language Tasks](https://mlsanity.com/paper/2208.10758/) (2022)
- [Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning](https://mlsanity.com/paper/2606.10533/) (2026)
- [GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation](https://mlsanity.com/paper/2406.12834/) (2024)
- [Harnessing Vision-Language Pretrained Models with Temporal-Aware Adaptation for Referring Video Object Segmentation](https://mlsanity.com/paper/2405.10610/) (2024)
- [Temporal-Conditional Referring Video Object Segmentation with Noise-Free Text-to-Video Diffusion Model](https://mlsanity.com/paper/2508.13584/) (2025)
- [Geometry-aware Gaussian Prior and Axial Attention for Cervical Cytology Image Classification](https://mlsanity.com/paper/2607.10278/) (2026)
- [Mitigating Query Selection Bias in Referring Video Object Segmentation](https://mlsanity.com/paper/2509.13722/) (2025)
- [LTCA: Long-range Temporal Context Attention for Referring Video Object Segmentation](https://mlsanity.com/paper/2510.08305/) (2025)
- [Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection](https://mlsanity.com/paper/2607.18693/) (2026)
- [DSPO: Diversity-aware Subjective Policy Optimization for Robust Emotional Reasoning](https://mlsanity.com/paper/2609.36775/) (2026)
- [Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning](https://mlsanity.com/paper/2609.36782/) (2026)
- [FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models](https://mlsanity.com/paper/2610.16744/) (2026)

## Abstract (arXiv)

Referring video object segmentation aims to segment the object referred by a given language expression. Existing works typically require compressed video bitstream to be decoded to RGB frames before being segmented, which increases computation and storage requirements and ultimately slows the inference down. This may hamper its application in real-world computing resource limited scenarios, such as autonomous cars and drones. To alleviate this problem, in this paper, we explore the referring object segmentation task on compressed videos, namely on the original video data flow. Besides the inherent difficulty of the video referring object segmentation task itself, obtaining discriminative representation from compressed video is also rather challenging. To address this problem, we propose a multi-attention network which consists of dual-path dual-attention module and a query-based cross-modal Transformer module. Specifically, the dual-path dual-attention module is designed to extract effective representation from compressed data in three modalities, i.e., I-frame, Motion Vector and Residual. The query-based cross-modal Transformer firstly models the correlation between linguistic and visual modalities, and then the fused multi-modality features are used to guide object queries to generate a content-aware dynamic kernel and to predict final segmentation masks. Different from previous works, we propose to learn just one kernel, which thus removes the complicated post mask-matching procedure of existing methods. Extensive promising experimental results on three challenging datasets show the effectiveness of our method compared against several state-of-the-art methods which are proposed for processing RGB data. Source code is available at: https://github.com/DexiangHong/MANet.