# Flow-Guided Video Inpainting with Scene Templates

arXiv: https://arxiv.org/abs/2108.12845 · HTML: https://mlsanity.com/paper/2108.12845/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 29 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [Deep Video Inpainting](https://mlsanity.com/paper/1905.01639/) (2019)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Unsupervised Moving Object Detection via Contextual Information Separation](https://mlsanity.com/paper/1901.03360/) (2019)
- [Video Inpainting by Jointly Learning Temporal Structure and Spatial Details](https://mlsanity.com/paper/1806.08482/) (2018)
- [Copy-and-Paste Networks for Deep Video Inpainting](https://mlsanity.com/paper/1908.11587/) (2019)
- [Onion-Peel Networks for Deep Video Completion](https://mlsanity.com/paper/1908.08718/) (2019)
- [Video Inpainting of Complex Scenes](https://mlsanity.com/paper/1503.05528/) (2015)
- [Short-Term and Long-Term Context Aggregation Network for Video Inpainting](https://mlsanity.com/paper/2009.05721/) (2020)
- [An Internal Learning Approach to Video Inpainting](https://mlsanity.com/paper/1909.07957/) (2019)

## Built on by (most cited first, 12 of 12)

- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2207.08485/) (2022)
- [Bitstream-Corrupted Video Recovery: A Novel Benchmark Dataset and Method](https://mlsanity.com/paper/2309.13890/) (2023)
- [Co-attention Propagation Network for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2304.03910/) (2023)
- [Semi-Supervised Video Inpainting with Cycle Consistency Constraints](https://mlsanity.com/paper/2208.06807/) (2022)
- [Diffeomorphic Template Registration for Atmospheric Turbulence Mitigation](https://mlsanity.com/paper/2405.03662/) (2024)
- [Blind Bitstream-corrupted Video Recovery via Metadata-guided Diffusion Model](https://mlsanity.com/paper/2604.13906/) (2026)
- [Triangular Consistency as a Universal Constraint for Learning Optical Flow](https://mlsanity.com/paper/2606.19938/) (2026)
- [On the Viability of Monocular Depth Pre-training for Semantic Segmentation](https://mlsanity.com/paper/2203.13987/) (2022)
- [Reimagining Reality: A Comprehensive Survey of Video Inpainting Techniques](https://mlsanity.com/paper/2401.17883/) (2024)
- [Optimised ProPainter for Video Diminished Reality Inpainting](https://mlsanity.com/paper/2406.02287/) (2024)

## Abstract (arXiv)

We consider the problem of filling in missing spatio-temporal regions of a video. We provide a novel flow-based solution by introducing a generative model of images in relation to the scene (without missing regions) and mappings from the scene to images. We use the model to jointly infer the scene template, a 2D representation of the scene, and the mappings. This ensures consistency of the frame-to-frame flows generated to the underlying scene, reducing geometric distortions in flow based inpainting. The template is mapped to the missing regions in the video by a new L2-L1 interpolation scheme, creating crisp inpaintings and reducing common blur and distortion artifacts. We show on two benchmark datasets that our approach out-performs state-of-the-art quantitatively and in user studies.