# Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs

arXiv: https://arxiv.org/abs/2402.09100 · HTML: https://mlsanity.com/paper/2402.09100/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Builds on (most cited first, 17 of 17)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces](https://mlsanity.com/paper/1803.09179/) (2018)
- [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Video Inpainting by Jointly Learning Temporal Structure and Spatial Details](https://mlsanity.com/paper/1806.08482/) (2018)
- [Progressive Temporal Feature Alignment Network for Video Inpainting](https://mlsanity.com/paper/2104.03507/) (2021)
- [A survey on GANs for computer vision: Recent research, analysis and taxonomy](https://mlsanity.com/paper/2203.11242/) (2022)
- [The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting](https://mlsanity.com/paper/2105.05332/) (2021)
- [Deep Face Video Inpainting via UV Mapping](https://mlsanity.com/paper/2109.00681/) (2021)
- [Expression-aware video inpainting for HMD removal in XR applications](https://mlsanity.com/paper/2401.14136/) (2024)

## Abstract (arXiv)

Facial video inpainting plays a crucial role in a wide range of applications, including but not limited to the removal of obstructions in video conferencing and telemedicine, enhancement of facial expression analysis, privacy protection, integration of graphical overlays, and virtual makeup. This domain presents serious challenges due to the intricate nature of facial features and the inherent human familiarity with faces, heightening the need for accurate and persuasive completions. In addressing challenges specifically related to occlusion removal in this context, our focus is on the progressive task of generating complete images from facial data covered by masks, ensuring both spatial and temporal coherence. Our study introduces a network designed for expression-based video inpainting, employing generative adversarial networks (GANs) to handle static and moving occlusions across all frames. By utilizing facial landmarks and an occlusion-free reference image, our model maintains the user's identity consistently across frames. We further enhance emotional preservation through a customized facial expression recognition (FER) loss function, ensuring detailed inpainted outputs. Our proposed framework exhibits proficiency in eliminating occlusions from facial videos in an adaptive form, whether appearing static or dynamic on the frames, while providing realistic and coherent results.