# Video Instance Matting

arXiv: https://arxiv.org/abs/2311.04212 · HTML: https://mlsanity.com/paper/2311.04212/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 53 references that are in the map.
- First posted on arXiv 7 Nov 2023.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition](https://mlsanity.com/paper/2011.11961/) (2020)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation](https://mlsanity.com/paper/2007.14772/) (2020)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [Bridging Composite and Real: Towards End-to-end Deep Image Matting](https://mlsanity.com/paper/2010.16188/) (2020)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)
- [Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation](https://mlsanity.com/paper/1909.09725/) (2019)
- [Natural Image Matting via Guided Contextual Attention](https://mlsanity.com/paper/2001.04069/) (2020)
- [Semantic Human Matting](https://mlsanity.com/paper/1809.01354/) (2018)
- [Information-Flow Matting](https://mlsanity.com/paper/1707.05055/) (2017)
- [SeMask: Semantically Masked Transformers for Semantic Segmentation](https://mlsanity.com/paper/2112.12782/) (2021)
- [F, B, Alpha Matting](https://mlsanity.com/paper/2003.07711/) (2020)
- [Boosting Semantic Human Matting with Coarse Annotations](https://mlsanity.com/paper/2004.04955/) (2020)
- [Deep Video Matting via Spatio-Temporal Alignment and Aggregation](https://mlsanity.com/paper/2104.11208/) (2021)
- [High-Resolution Deep Image Matting](https://mlsanity.com/paper/2009.06613/) (2020)
- [Matting Anything](https://mlsanity.com/paper/2306.05399/) (2023)

## Built on by (most cited first, 4 of 4)

- [MatAnyone: Stable Video Matting with Consistent Memory Propagation](https://mlsanity.com/paper/2501.14677/) (2025)
- [MaGGIe: Masked Guided Gradual Human Instance Matting](https://mlsanity.com/paper/2404.16035/) (2024)
- [MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluator](https://mlsanity.com/paper/2512.11782/) (2025)
- [VRMDiff: Text-Guided Video Referring Matting Generation of Diffusion](https://mlsanity.com/paper/2503.10678/) (2025)

## Abstract (arXiv)

Conventional video matting outputs one alpha matte for all instances appearing in a video frame so that individual instances are not distinguished. While video instance segmentation provides time-consistent instance masks, results are unsatisfactory for matting applications, especially due to applied binarization. To remedy this deficiency, we propose Video Instance Matting~(VIM), that is, estimating alpha mattes of each instance at each frame of a video sequence. To tackle this challenging problem, we present MSG-VIM, a Mask Sequence Guided Video Instance Matting neural network, as a novel baseline model for VIM. MSG-VIM leverages a mixture of mask augmentations to make predictions robust to inaccurate and inconsistent mask guidance. It incorporates temporal mask and temporal feature guidance to improve the temporal consistency of alpha matte predictions. Furthermore, we build a new benchmark for VIM, called VIM50, which comprises 50 video clips with multiple human instances as foreground objects. To evaluate performances on the VIM task, we introduce a suitable metric called Video Instance-aware Matting Quality~(VIMQ). Our proposed model MSG-VIM sets a strong baseline on the VIM50 benchmark and outperforms existing methods by a large margin. The project is open-sourced at https://github.com/SHI-Labs/VIM.