# DVIS: Decoupled Video Instance Segmentation Framework

arXiv: https://arxiv.org/abs/2306.03413 · HTML: https://mlsanity.com/paper/2306.03413/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Occluded Video Instance Segmentation: A Benchmark](https://mlsanity.com/paper/2102.01558/) (2021)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation](https://mlsanity.com/paper/2007.14772/) (2020)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [SeqFormer: Sequential Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2112.08275/) (2021)
- [In Defense of Online Models for Video Instance Segmentation](https://mlsanity.com/paper/2207.10661/) (2022)
- [ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2012.05258/) (2020)
- [VITA: Video Instance Segmentation via Object Token Association](https://mlsanity.com/paper/2206.04403/) (2022)
- [MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training](https://mlsanity.com/paper/2208.02245/) (2022)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)
- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [A Generalized Framework for Video Instance Segmentation](https://mlsanity.com/paper/2211.08834/) (2022)
- [Efficient Video Instance Segmentation via Tracklet Query and Proposal](https://mlsanity.com/paper/2203.01853/) (2022)
- [TarViS: A Unified Approach for Target-based Video Segmentation](https://mlsanity.com/paper/2301.02657/) (2023)
- [VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation](https://mlsanity.com/paper/2112.04177/) (2021)
- [Learning to Associate Every Segment for Video Panoptic Segmentation](https://mlsanity.com/paper/2106.09453/) (2021)
- [Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentation](https://mlsanity.com/paper/2304.04694/) (2023)
- [MDQE: Mining Discriminative Query Embeddings to Segment Occluded Instances on Challenging Videos](https://mlsanity.com/paper/2303.14395/) (2023)
- [Robust Online Video Instance Segmentation with Track Queries](https://mlsanity.com/paper/2211.09108/) (2022)
- [ISDA: Position-Aware Instance Segmentation with Deformable Attention](https://mlsanity.com/paper/2202.12251/) (2022)

## Built on by (most cited first, 30 of 77)

- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [VISA: Reasoning Video Object Segmentation via Large Language Models](https://mlsanity.com/paper/2407.11325/) (2024)
- [General Object Foundation Model for Images and Videos at Scale](https://mlsanity.com/paper/2312.09158/) (2023)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [CTVIS: Consistent Training for Online Video Instance Segmentation](https://mlsanity.com/paper/2307.12616/) (2023)
- [UniVS: Unified and Universal Video Segmentation with Prompts as Queries](https://mlsanity.com/paper/2402.18115/) (2024)
- [ViLLa: Video Reasoning Segmentation with Large Language Model](https://mlsanity.com/paper/2407.14500/) (2024)
- [Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding](https://mlsanity.com/paper/2504.10465/) (2025)
- [Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2310.18709/) (2023)
- [Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2501.04670/) (2025)
- [SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment](https://mlsanity.com/paper/2507.02705/) (2025)
- [Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2407.07427/) (2024)
- [PVUW 2024 Challenge on Complex Video Understanding: Methods and Results](https://mlsanity.com/paper/2406.17005/) (2024)
- [CAVIS: Context-Aware Video Instance Segmentation](https://mlsanity.com/paper/2407.03010/) (2024)
- [Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation](https://mlsanity.com/paper/2507.22886/) (2025)
- [CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos](https://mlsanity.com/paper/2505.18561/) (2025)
- [Dense360: Dense Understanding from Omnidirectional Panoramas](https://mlsanity.com/paper/2506.14471/) (2025)
- [LSVOS Challenge Report: Large-scale Complex and Long Video Object Segmentation](https://mlsanity.com/paper/2409.05847/) (2024)
- [VidEoMT: Your ViT is Secretly Also a Video Segmentation Model](https://mlsanity.com/paper/2602.17807/) (2026)
- [1st Place Solution for PVUW Challenge 2023: Video Panoptic Segmentation](https://mlsanity.com/paper/2306.04091/) (2023)
- [VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement](https://mlsanity.com/paper/2312.04885/) (2023)
- [SMITE: Segment Me In TimE](https://mlsanity.com/paper/2410.18538/) (2024)
- [OnlineTAS: An Online Baseline for Temporal Action Segmentation](https://mlsanity.com/paper/2411.01122/) (2024)
- [The 1st Solution for 7th LSVOS RVOS Track: SaSaSa2VA](https://mlsanity.com/paper/2509.16972/) (2025)
- [1st Place Solution for the 5th LSVOS Challenge: Video Instance Segmentation](https://mlsanity.com/paper/2308.14392/) (2023)
- [RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything](https://mlsanity.com/paper/2401.10228/) (2024)
- [General and Task-Oriented Video Segmentation](https://mlsanity.com/paper/2407.06540/) (2024)
- [NSM4D: Neural Scene Model Based Online 4D Point Cloud Sequence Understanding](https://mlsanity.com/paper/2310.08326/) (2023)
- [SyncVIS: Synchronized Video Instance Segmentation](https://mlsanity.com/paper/2412.00882/) (2024)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)

## Abstract (arXiv)

Video instance segmentation (VIS) is a critical task with diverse applications, including autonomous driving and video editing. Existing methods often underperform on complex and long videos in real world, primarily due to two factors. Firstly, offline methods are limited by the tightly-coupled modeling paradigm, which treats all frames equally and disregards the interdependencies between adjacent frames. Consequently, this leads to the introduction of excessive noise during long-term temporal alignment. Secondly, online methods suffer from inadequate utilization of temporal information. To tackle these challenges, we propose a decoupling strategy for VIS by dividing it into three independent sub-tasks: segmentation, tracking, and refinement. The efficacy of the decoupling strategy relies on two crucial elements: 1) attaining precise long-term alignment outcomes via frame-by-frame association during tracking, and 2) the effective utilization of temporal information predicated on the aforementioned accurate alignment outcomes during refinement. We introduce a novel referring tracker and temporal refiner to construct the Decoupled VIS framework (DVIS). DVIS achieves new SOTA performance in both VIS and VPS, surpassing the current SOTA methods by 7.3 AP and 9.6 VPQ on the OVIS and VIPSeg datasets, which are the most challenging and realistic benchmarks. Moreover, thanks to the decoupling strategy, the referring tracker and temporal refiner are super light-weight (only 1.69% of the segmenter FLOPs), allowing for efficient training and inference on a single GPU with 11G memory. The code is available at hrefhttps://github.com/zhang-tao-whu/DVIShttps://github.com/zhang-tao-whu/DVIS.