# Deep Video Matting via Spatio-Temporal Alignment and Aggregation

arXiv: https://arxiv.org/abs/2104.11208 · HTML: https://mlsanity.com/paper/2104.11208/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 48 references that are in the map.
- First posted on arXiv 22 Apr 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network](https://mlsanity.com/paper/1703.02719/) (2017)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [Indices Matter: Learning to Index for Deep Image Matting](https://mlsanity.com/paper/1908.00672/) (2019)
- [Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation](https://mlsanity.com/paper/1909.09725/) (2019)
- [Natural Image Matting via Guided Contextual Attention](https://mlsanity.com/paper/2001.04069/) (2020)
- [Semantic Human Matting](https://mlsanity.com/paper/1809.01354/) (2018)
- [Information-Flow Matting](https://mlsanity.com/paper/1707.05055/) (2017)
- [AlphaGAN: Generative adversarial networks for natural image matting](https://mlsanity.com/paper/1807.10088/) (2018)
- [Boosting Semantic Human Matting with Coarse Annotations](https://mlsanity.com/paper/2004.04955/) (2020)
- [Fast Deep Matting for Portrait Animation on Mobile Phone](https://mlsanity.com/paper/1707.08289/) (2017)

## Built on by (most cited first, 30 of 41)

- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [Matting Anything](https://mlsanity.com/paper/2306.05399/) (2023)
- [Attention-guided Temporally Coherent Video Object Matting](https://mlsanity.com/paper/2105.11427/) (2021)
- [Human Instance Matting via Mutual Guidance and Multi-Instance Refinement](https://mlsanity.com/paper/2205.10767/) (2022)
- [MatAnyone: Stable Video Matting with Consistent Memory Propagation](https://mlsanity.com/paper/2501.14677/) (2025)
- [VMFormer: End-to-End Video Matting with Transformer](https://mlsanity.com/paper/2208.12801/) (2022)
- [MaGGIe: Masked Guided Gradual Human Instance Matting](https://mlsanity.com/paper/2404.16035/) (2024)
- [One-Trimap Video Matting](https://mlsanity.com/paper/2207.13353/) (2022)
- [Deep Image Matting: A Comprehensive Survey](https://mlsanity.com/paper/2304.04672/) (2023)
- [Deep Video Harmonization with Color Mapping Consistency](https://mlsanity.com/paper/2205.00687/) (2022)
- [Co-attention Propagation Network for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2304.03910/) (2023)
- [FactorMatte: Redefining Video Matting for Re-Composition Tasks](https://mlsanity.com/paper/2211.02145/) (2022)
- [PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion](https://mlsanity.com/paper/2606.01399/) (2026)
- [MODNet-V: Improving Portrait Video Matting via Background Restoration](https://mlsanity.com/paper/2109.11818/) (2021)
- [Adaptive Human Matting for Dynamic Videos](https://mlsanity.com/paper/2304.06018/) (2023)
- [Video Generation with Stable Transparency via Shiftable RGB-A Distribution Learner](https://mlsanity.com/paper/2509.24979/) (2025)
- [MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluator](https://mlsanity.com/paper/2512.11782/) (2025)
- [EFormer: Enhanced Transformer towards Semantic-Contour Features of Foreground for Portraits Matting](https://mlsanity.com/paper/2308.12831/) (2023)
- [Video Instance Matting](https://mlsanity.com/paper/2311.04212/) (2023)
- [Boosting General Trimap-free Matting in the Real-World Image](https://mlsanity.com/paper/2405.17916/) (2024)
- [VideoMaMa: Mask-Guided Video Matting via Generative Prior](https://mlsanity.com/paper/2601.14255/) (2026)
- [Alpha Matte Generation from Single Input for Portrait Matting](https://mlsanity.com/paper/2106.03210/) (2021)
- [Towards Label-Efficient Human Matting: A Simple Baseline for Weakly Semi-Supervised Trimap-Free Human Matting](https://mlsanity.com/paper/2404.00921/) (2024)
- [Efficient Heterogeneous Video Segmentation at the Edge](https://mlsanity.com/paper/2208.11666/) (2022)
- [Tsanet: Temporal and Scale Alignment for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2303.04376/) (2023)
- [Object-Aware Video Matting with Cross-Frame Guidance](https://mlsanity.com/paper/2503.01262/) (2025)
- [VRMDiff: Text-Guided Video Referring Matting Generation of Diffusion](https://mlsanity.com/paper/2503.10678/) (2025)
- [Color-aware Deep Temporal Backdrop Duplex Matting System](https://mlsanity.com/paper/2306.02954/) (2023)
- [BiVM: Accurate Binarized Neural Network for Efficient Video Matting](https://mlsanity.com/paper/2507.04456/) (2025)
- [EasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decomposition](https://mlsanity.com/paper/2512.21865/) (2025)

## Abstract (arXiv)

Despite the significant progress made by deep learning in natural image matting, there has been so far no representative work on deep learning for video matting due to the inherent technical challenges in reasoning temporal domain and lack of large-scale video matting datasets. In this paper, we propose a deep learning-based video matting framework which employs a novel and effective spatio-temporal feature aggregation module (ST-FAM). As optical flow estimation can be very unreliable within matting regions, ST-FAM is designed to effectively align and aggregate information across different spatial scales and temporal frames within the network decoder. To eliminate frame-by-frame trimap annotations, a lightweight interactive trimap propagation network is also introduced. The other contribution consists of a large-scale video matting dataset with groundtruth alpha mattes for quantitative evaluation and real-world high-resolution videos with trimaps for qualitative evaluation. Quantitative and qualitative experimental results show that our framework significantly outperforms conventional video matting and deep image matting methods applied to video in presence of multi-frame temporal information.