# Robust High-Resolution Video Matting with Temporal Guidance

arXiv: https://arxiv.org/abs/2108.11515 · HTML: https://mlsanity.com/paper/2108.11515/

## Key facts

- Cited by 137 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 25 Aug 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [RVOS: End-to-End Recurrent Network for Video Object Segmentation](https://mlsanity.com/paper/1903.05612/) (2019)
- [Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation](https://mlsanity.com/paper/1909.09725/) (2019)
- [Learning Video Object Segmentation with Visual Memory](https://mlsanity.com/paper/1704.05737/) (2017)
- [Information-Flow Matting](https://mlsanity.com/paper/1707.05055/) (2017)
- [Fast End-to-End Trainable Guided Filter](https://mlsanity.com/paper/1803.05619/) (2018)
- [F, B, Alpha Matting](https://mlsanity.com/paper/2003.07711/) (2020)
- [Deep Video Matting via Spatio-Temporal Alignment and Aggregation](https://mlsanity.com/paper/2104.11208/) (2021)
- [Fast Deep Matting for Portrait Animation on Mobile Phone](https://mlsanity.com/paper/1707.08289/) (2017)
- [Semantic Segmentation of Video Sequences with Convolutional LSTMs](https://mlsanity.com/paper/1905.01058/) (2019)

## Built on by (most cited first, 30 of 137)

- [SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting](https://mlsanity.com/paper/2403.05087/) (2024)
- [HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion](https://mlsanity.com/paper/2305.06356/) (2023)
- [Instant Volumetric Head Avatars](https://mlsanity.com/paper/2211.12499/) (2022)
- [Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition](https://mlsanity.com/paper/2302.11566/) (2023)
- [NeRF-Art: Text-Driven Neural Radiance Fields Stylization](https://mlsanity.com/paper/2212.08070/) (2022)
- [Highly Accurate Dichotomous Image Segmentation](https://mlsanity.com/paper/2203.03041/) (2022)
- [FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embedding](https://mlsanity.com/paper/2312.02214/) (2023)
- [EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild](https://mlsanity.com/paper/2308.16894/) (2023)
- [4K4D: Real-Time 4D View Synthesis at 4K Resolution](https://mlsanity.com/paper/2310.11448/) (2023)
- [Hi4D: 4D Instance Segmentation of Close Human Interaction](https://mlsanity.com/paper/2303.15380/) (2023)
- [MonoHuman: Animatable Human Neural Field from Monocular Video](https://mlsanity.com/paper/2304.02001/) (2023)
- [AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels](https://mlsanity.com/paper/2211.13206/) (2022)
- [Harmonizer: Learning to Perform White-Box Image and Video Harmonization](https://mlsanity.com/paper/2207.01322/) (2022)
- [HeadGaS: Real-Time Animatable Head Avatars via 3D Gaussian Splatting](https://mlsanity.com/paper/2312.02902/) (2023)
- [Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera](https://mlsanity.com/paper/2206.15258/) (2022)
- [ViTMatte: Boosting Image Matting with Pretrained Plain Vision Transformers](https://mlsanity.com/paper/2305.15272/) (2023)
- [MVHumanNet: A Large-scale Dataset of Multi-view Daily Dressing Human Captures](https://mlsanity.com/paper/2312.02963/) (2023)
- [StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Video](https://mlsanity.com/paper/2305.00942/) (2023)
- [CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models](https://mlsanity.com/paper/2412.12093/) (2024)
- [LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar](https://mlsanity.com/paper/2305.01190/) (2023)
- [AutoDecoding Latent 3D Diffusion Models](https://mlsanity.com/paper/2307.05445/) (2023)
- [AnimateDiff-Lightning: Cross-Model Diffusion Distillation](https://mlsanity.com/paper/2403.12706/) (2024)
- [RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars](https://mlsanity.com/paper/2305.13353/) (2023)
- [GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion](https://mlsanity.com/paper/2412.10209/) (2024)
- [Matting Anything](https://mlsanity.com/paper/2306.05399/) (2023)
- [SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage](https://mlsanity.com/paper/2204.03648/) (2022)
- [RGBAvatar: Reduced Gaussian Blendshapes for Online Modeling of Head Avatars](https://mlsanity.com/paper/2503.12886/) (2025)
- [HARP: Personalized Hand Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2212.09530/) (2022)
- [Relightable and Animatable Neural Avatar from Sparse-View Video](https://mlsanity.com/paper/2308.07903/) (2023)
- [FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction](https://mlsanity.com/paper/2206.02194/) (2022)

## Abstract (arXiv)

We introduce a robust, real-time, high-resolution human video matting method that achieves new state-of-the-art performance. Our method is much lighter than previous approaches and can process 4K at 76 FPS and HD at 104 FPS on an Nvidia GTX 1080Ti GPU. Unlike most existing methods that perform video matting frame-by-frame as independent images, our method uses a recurrent architecture to exploit temporal information in videos and achieves significant improvements in temporal coherence and matting quality. Furthermore, we propose a novel training strategy that enforces our network on both matting and segmentation objectives. This significantly improves our model's robustness. Our method does not require any auxiliary inputs such as a trimap or a pre-captured background image, so it can be widely applied to existing human matting applications.