# MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation arXiv: https://arxiv.org/abs/2501.13667 ยท HTML: https://mlsanity.com/paper/2501.13667/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 45 of its 61 references that are in the map. - First posted on arXiv 23 Jan 2025. ## Builds on (most cited first, 30 of 45) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024) - [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016) - [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016) - [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017) - [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015) - [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023) - [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023) - [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022) - [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019) - [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023) - [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023) - [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021) - [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023) - [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023) - [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023) - [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023) - [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022) - [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018) - [End-to-End Referring Video Object Segmentation with Multimodal Transformers](https://mlsanity.com/paper/2111.14821/) (2021) - [VISA: Reasoning Video Object Segmentation via Large Language Models](https://mlsanity.com/paper/2407.11325/) (2024) - [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023) - [EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model](https://mlsanity.com/paper/2406.20076/) (2024) - [OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation](https://mlsanity.com/paper/2307.09356/) (2023) - [SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation](https://mlsanity.com/paper/2305.17011/) (2023) ## Built on by (most cited first, 7 of 7) - [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025) - [RS2-SAM2: Customized SAM2 for Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2503.07266/) (2025) - [MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding](https://mlsanity.com/paper/2510.09274/) (2025) - [SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation](https://mlsanity.com/paper/2603.18086/) (2026) - [VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation](https://mlsanity.com/paper/2603.27060/) (2026) - [CroBIM-V: Memory-Quality Controlled Remote Sensing Referring Video Object Segmentation](https://mlsanity.com/paper/2601.12076/) (2026) - [FoundYou: A Unified Model for Personalized Segmentation and Retrieval](https://mlsanity.com/paper/2608.29917/) (2026) ## Abstract (arXiv) Referring video object segmentation (RVOS) aims to segment objects in a video according to textual descriptions, which requires the integration of multimodal information and temporal dynamics perception. The Segment Anything Model 2 (SAM 2) has shown great effectiveness across various video segmentation tasks. However, its application to offline RVOS is challenged by the translation of the text into effective prompts and a lack of global context awareness. In this paper, we propose a novel RVOS framework, termed MPG-SAM 2, to address these challenges. Specifically, MPG-SAM 2 employs a unified multimodal encoder to jointly encode video and textual features, generating semantically aligned video and text embeddings, along with multimodal class tokens. A mask prior generator utilizes the video embeddings and class tokens to create pseudo masks of target objects and global context. These masks are fed into the prompt encoder as dense prompts along with multimodal class tokens as sparse prompts to generate accurate prompts for SAM 2. To provide the online SAM 2 with a global view, we introduce a hierarchical global-historical aggregator, which allows SAM 2 to aggregate global and historical information of target objects at both pixel and object levels, enhancing the target representation and temporal consistency. Extensive experiments on several RVOS benchmarks demonstrate the superiority of MPG-SAM 2 and the effectiveness of our proposed modules. The code is available at https://github.com/rongfu-dsb/MPG-SAM2.