# VMFormer: End-to-End Video Matting with Transformer

arXiv: https://arxiv.org/abs/2208.12801 · HTML: https://mlsanity.com/paper/2208.12801/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 54 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Neighborhood Attention Transformer](https://mlsanity.com/paper/2204.07143/) (2022)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [Mask Guided Matting via Progressive Refinement Network](https://mlsanity.com/paper/2012.06722/) (2020)
- [Natural Image Matting via Guided Contextual Attention](https://mlsanity.com/paper/2001.04069/) (2020)
- [Semantic Human Matting](https://mlsanity.com/paper/1809.01354/) (2018)
- [Information-Flow Matting](https://mlsanity.com/paper/1707.05055/) (2017)
- [Deep Automatic Natural Image Matting](https://mlsanity.com/paper/2107.07235/) (2021)
- [SeMask: Semantically Masked Transformers for Semantic Segmentation](https://mlsanity.com/paper/2112.12782/) (2021)
- [F, B, Alpha Matting](https://mlsanity.com/paper/2003.07711/) (2020)

## Built on by (most cited first, 20 of 20)

- [CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts](https://mlsanity.com/paper/2405.05949/) (2024)
- [Matting Anything](https://mlsanity.com/paper/2306.05399/) (2023)
- [MatAnyone: Stable Video Matting with Consistent Memory Propagation](https://mlsanity.com/paper/2501.14677/) (2025)
- [MaGGIe: Masked Guided Gradual Human Instance Matting](https://mlsanity.com/paper/2404.16035/) (2024)
- [Adaptive Human Matting for Dynamic Videos](https://mlsanity.com/paper/2304.06018/) (2023)
- [MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluator](https://mlsanity.com/paper/2512.11782/) (2025)
- [Video Instance Matting](https://mlsanity.com/paper/2311.04212/) (2023)
- [VideoMaMa: Mask-Guided Video Matting via Generative Prior](https://mlsanity.com/paper/2601.14255/) (2026)
- [UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors](https://mlsanity.com/paper/2605.00658/) (2026)
- [CineMatte: Background Matting for Virtual Production and Beyond](https://mlsanity.com/paper/2605.18328/) (2026)
- [ILDiff: Generate Transparent Animated Stickers by Implicit Layout Distillation](https://mlsanity.com/paper/2412.20901/) (2024)
- [Object-Aware Video Matting with Cross-Frame Guidance](https://mlsanity.com/paper/2503.01262/) (2025)
- [VRMDiff: Text-Guided Video Referring Matting Generation of Diffusion](https://mlsanity.com/paper/2503.10678/) (2025)
- [Efficient Portrait Matte Creation With Layer Diffusion and Connectivity Priors](https://mlsanity.com/paper/2501.16147/) (2025)
- [BiVM: Accurate Binarized Neural Network for Efficient Video Matting](https://mlsanity.com/paper/2507.04456/) (2025)
- [TransText: Alpha-as-RGB Representation for Transparent Text Animation](https://mlsanity.com/paper/2603.17944/) (2026)
- [POBEVM: Real-time Video Matting via Progressively Optimize the Target Body and Edge](https://mlsanity.com/paper/2402.09731/) (2024)
- [Post-Training Quantization for Video Matting](https://mlsanity.com/paper/2506.10840/) (2025)
- [SAM2Matting: Generalized Image and Video Matting](https://mlsanity.com/paper/2606.27339/) (2026)
- [Parallax Portrait Matting](https://mlsanity.com/paper/2607.11205/) (2026)

## Abstract (arXiv)

Video matting aims to predict the alpha mattes for each frame from a given input video sequence. Recent solutions to video matting have been dominated by deep convolutional neural networks (CNN) for the past few years, which have become the de-facto standard for both academia and industry. However, they have inbuilt inductive bias of locality and do not capture global characteristics of an image due to the CNN-based architectures. They also lack long-range temporal modeling considering computational costs when dealing with feature maps of multiple frames. In this paper, we propose VMFormer: a transformer-based end-to-end method for video matting. It makes predictions on alpha mattes of each frame from learnable queries given a video input sequence. Specifically, it leverages self-attention layers to build global integration of feature sequences with short-range temporal modeling on successive frames. We further apply queries to learn global representations through cross-attention in the transformer decoder with long-range temporal modeling upon all queries. In the prediction stage, both queries and corresponding feature maps are used to make the final prediction of alpha matte. Experiments show that VMFormer outperforms previous CNN-based video matting methods on the composited benchmarks. To our best knowledge, it is the first end-to-end video matting solution built upon a full vision transformer with predictions on the learnable queries. The project is open-sourced at https://chrisjuniorli.github.io/project/VMFormer/