# Deep Face Video Inpainting via UV Mapping

arXiv: https://arxiv.org/abs/2109.00681 · HTML: https://mlsanity.com/paper/2109.00681/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 94 references that are in the map.
- First posted on arXiv 2 Sep 2021.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [StructureFlow: Image Inpainting via Structure-aware Appearance Flow](https://mlsanity.com/paper/1908.03852/) (2019)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [Deep Video Inpainting](https://mlsanity.com/paper/1905.01639/) (2019)
- [SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color](https://mlsanity.com/paper/1902.06838/) (2019)
- [Blind Face Restoration via Deep Multi-scale Component Dictionaries](https://mlsanity.com/paper/2008.00418/) (2020)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [Progressive Semantic-Aware Style Transformation for Blind Face Restoration](https://mlsanity.com/paper/2009.08709/) (2020)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting](https://mlsanity.com/paper/1904.07475/) (2019)
- [High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis](https://mlsanity.com/paper/1611.09969/) (2016)
- [Foreground-aware Image Inpainting](https://mlsanity.com/paper/1901.05945/) (2019)
- [Shift-Net: Image Inpainting via Deep Feature Rearrangement](https://mlsanity.com/paper/1801.09392/) (2018)
- [Learning Warped Guidance for Blind Face Restoration](https://mlsanity.com/paper/1804.04829/) (2018)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Generative Face Completion](https://mlsanity.com/paper/1704.05838/) (2017)

## Built on by (most cited first, 7 of 7)

- [DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generation](https://mlsanity.com/paper/2203.07931/) (2022)
- [Survey on Deep Face Restoration: From Non-blind to Blind and Beyond](https://mlsanity.com/paper/2309.15490/) (2023)
- [Geometric Prior Based Deep Human Point Cloud Geometry Compression](https://mlsanity.com/paper/2305.01309/) (2023)
- [MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance](https://mlsanity.com/paper/2504.21497/) (2025)
- [Expression-aware video inpainting for HMD removal in XR applications](https://mlsanity.com/paper/2401.14136/) (2024)
- [Raformer: Redundancy-Aware Transformer for Video Wire Inpainting](https://mlsanity.com/paper/2404.15802/) (2024)
- [Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs](https://mlsanity.com/paper/2402.09100/) (2024)

## Abstract (arXiv)

This paper addresses the problem of face video inpainting. Existing video inpainting methods target primarily at natural scenes with repetitive patterns. They do not make use of any prior knowledge of the face to help retrieve correspondences for the corrupted face. They therefore only achieve sub-optimal results, particularly for faces under large pose and expression variations where face components appear very differently across frames. In this paper, we propose a two-stage deep learning method for face video inpainting. We employ 3DMM as our 3D face prior to transform a face between the image space and the UV (texture) space. In Stage I, we perform face inpainting in the UV space. This helps to largely remove the influence of face poses and expressions and makes the learning task much easier with well aligned face features. We introduce a frame-wise attention module to fully exploit correspondences in neighboring frames to assist the inpainting task. In Stage II, we transform the inpainted face regions back to the image space and perform face video refinement that inpaints any background regions not covered in Stage I and also refines the inpainted face regions. Extensive experiments have been carried out which show our method can significantly outperform methods based merely on 2D information, especially for faces under large pose and expression variations. Project page: https://ywq.github.io/FVIP