# Deep Learning-based Image and Video Inpainting: A Survey

arXiv: https://arxiv.org/abs/2401.03395 · HTML: https://mlsanity.com/paper/2401.03395/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 161 of its 273 references that are in the map.
- First posted on arXiv 7 Jan 2024.

## Builds on (most cited first, 30 of 161)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)

## Built on by (most cited first, 30 of 44)

- [BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion](https://mlsanity.com/paper/2403.06976/) (2024)
- [A Survey on Diffusion Models for Inverse Problems](https://mlsanity.com/paper/2410.00083/) (2024)
- [VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control](https://mlsanity.com/paper/2503.05639/) (2025)
- [InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior](https://mlsanity.com/paper/2404.11613/) (2024)
- [BrushEdit: All-In-One Image Inpainting and Editing](https://mlsanity.com/paper/2412.10316/) (2024)
- [A Survey of Multimodal Hallucination Evaluation and Detection](https://mlsanity.com/paper/2507.19024/) (2025)
- [RORem: Training a Robust Object Remover with Human-in-the-Loop](https://mlsanity.com/paper/2501.00740/) (2025)
- [Bayesian Experimental Design via Contrastive Diffusions](https://mlsanity.com/paper/2410.11826/) (2024)
- [Flux Already Knows -- Activating Subject-Driven Image Generation without Training](https://mlsanity.com/paper/2504.11478/) (2025)
- [KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution](https://mlsanity.com/paper/2505.00497/) (2025)
- [PISCO: Precise Video Instance Insertion with Sparse Control](https://mlsanity.com/paper/2602.08277/) (2026)
- [View-consistent Object Removal in Radiance Fields](https://mlsanity.com/paper/2408.02100/) (2024)
- [Transformer-based Image and Video Inpainting: Current Challenges and Future Directions](https://mlsanity.com/paper/2407.00226/) (2024)
- [Geometry-Aware Diffusion Models for Multiview Scene Inpainting](https://mlsanity.com/paper/2502.13335/) (2025)
- [Replace Anyone in Videos](https://mlsanity.com/paper/2409.19911/) (2024)
- [OutDreamer: Video Outpainting with a Diffusion Transformer](https://mlsanity.com/paper/2506.22298/) (2025)
- [Vision-Language Model Guided Image Restoration](https://mlsanity.com/paper/2512.17292/) (2025)
- [High-fidelity 3D Gaussian Inpainting: preserving multi-view consistency and photorealistic details](https://mlsanity.com/paper/2507.18023/) (2025)
- [Recursive InPainting (RIP): how much information is lost under recursive inferences?](https://mlsanity.com/paper/2407.09549/) (2024)
- [DreamMix: Decoupling Object Attributes for Enhanced Editability in Customized Image Inpainting](https://mlsanity.com/paper/2411.17223/) (2024)
- [Advances in Artificial Intelligence: A Review for the Creative Industries](https://mlsanity.com/paper/2501.02725/) (2025)
- [Creatively Upscaling Images with Global-Regional Priors](https://mlsanity.com/paper/2505.16976/) (2025)
- [Ultra High-Resolution Image Inpainting with Patch-Based Content Consistency Adapter](https://mlsanity.com/paper/2510.13419/) (2025)
- [CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal](https://mlsanity.com/paper/2603.21901/) (2026)
- [Fill in the ____ (a Diffusion-based Image Inpainting Pipeline)](https://mlsanity.com/paper/2403.16016/) (2024)
- [Local Lesion Generation is Effective for Capsule Endoscopy Image Data Augmentation in a Limited Data Setting](https://mlsanity.com/paper/2411.03098/) (2024)
- [PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control](https://mlsanity.com/paper/2412.01223/) (2024)
- [A Joint Learning Framework with Feature Reconstruction and Prediction for Incomplete Satellite Image Time Series in Agricultural Semantic Segmentation](https://mlsanity.com/paper/2505.19159/) (2025)
- [When and How to Cut Classical Concerts? A Multimodal Automated Video Editing Approach](https://mlsanity.com/paper/2510.05661/) (2025)
- [GLAD: Generative Language-Assisted Visual Tracking for Low-Semantic Templates](https://mlsanity.com/paper/2602.00570/) (2026)

## Abstract (arXiv)

Image and video inpainting is a classic problem in computer vision and computer graphics, aiming to fill in the plausible and realistic content in the missing areas of images and videos. With the advance of deep learning, this problem has achieved significant progress recently. The goal of this paper is to comprehensively review the deep learning-based methods for image and video inpainting. Specifically, we sort existing methods into different categories from the perspective of their high-level inpainting pipeline, present different deep learning architectures, including CNN, VAE, GAN, diffusion models, etc., and summarize techniques for module design. We review the training objectives and the common benchmark datasets. We present evaluation metrics for low-level pixel and high-level perceptional similarity, conduct a performance evaluation, and discuss the strengths and weaknesses of representative inpainting methods. We also discuss related real-world applications. Finally, we discuss open challenges and suggest potential future research directions.