# Flow-Guided Diffusion for Video Inpainting

arXiv: https://arxiv.org/abs/2311.15368 · HTML: https://mlsanity.com/paper/2311.15368/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 47 of its 54 references that are in the map.
- First posted on arXiv 26 Nov 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 47)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction](https://mlsanity.com/paper/2112.05146/) (2021)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting](https://mlsanity.com/paper/2212.06909/) (2022)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [Video Probabilistic Diffusion Models in Projected Latent Space](https://mlsanity.com/paper/2302.07685/) (2023)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation](https://mlsanity.com/paper/2305.10874/) (2023)
- [Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation](https://mlsanity.com/paper/2304.08477/) (2023)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)

## Built on by (most cited first, 11 of 11)

- [Video Diffusion Models are Strong Video Inpainter](https://mlsanity.com/paper/2408.11402/) (2024)
- [Optical-Flow Guided Prompt Optimization for Coherent Video Generation](https://mlsanity.com/paper/2411.15540/) (2024)
- [Semantically Consistent Video Inpainting with Conditional Diffusion Models](https://mlsanity.com/paper/2405.00251/) (2024)
- [Replace Anyone in Videos](https://mlsanity.com/paper/2409.19911/) (2024)
- [Video Virtual Try-on with Conditional Diffusion Transformer Inpainter](https://mlsanity.com/paper/2506.21270/) (2025)
- [V-LASIK: Consistent Glasses-Removal from Videos Using Synthetic Data](https://mlsanity.com/paper/2406.14510/) (2024)
- [End-to-End Learning-based Video Streaming Enhancement Pipeline: A Generative AI Approach](https://mlsanity.com/paper/2512.14185/) (2025)
- [GlobalPaint: Spatiotemporal Coherent Video Outpainting with Global Feature Guidance](https://mlsanity.com/paper/2601.06413/) (2026)
- [Reimagining Reality: A Comprehensive Survey of Video Inpainting Techniques](https://mlsanity.com/paper/2401.17883/) (2024)
- [VipDiff: Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models](https://mlsanity.com/paper/2501.12267/) (2025)
- [Perceptual Refinement of an End-to-End Video Streaming Pipeline via Generative AI Layers](https://mlsanity.com/paper/2609.19215/) (2026)

## Abstract (arXiv)

Video inpainting has been challenged by complex scenarios like large movements and low-light conditions. Current methods, including emerging diffusion models, face limitations in quality and efficiency. This paper introduces the Flow-Guided Diffusion model for Video Inpainting (FGDVI), a novel approach that significantly enhances temporal consistency and inpainting quality via reusing an off-the-shelf image generation diffusion model. We employ optical flow for precise one-step latent propagation and introduces a model-agnostic flow-guided latent interpolation technique. This technique expedites denoising, seamlessly integrating with any Video Diffusion Model (VDM) without additional training. Our FGDVI demonstrates a remarkable 10% improvement in flow warping error E_warp over existing state-of-the-art methods. Our comprehensive experiments validate superior performance of FGDVI, offering a promising direction for advanced video inpainting. The code and detailed results will be publicly available in https://github.com/NevSNev/FGDVI.