mlsanityData to 7 Oct 2026

Deep Learning-based Image and Video Inpainting: A Survey

arXiv 2401.03395 ↗First posted 7 Jan 2024cs.CVcited by 44

Not on the map. Why?

Abstract (arXiv)

Image and video inpainting is a classic problem in computer vision and computer graphics, aiming to fill in the plausible and realistic content in the missing areas of images and videos. With the advance of deep learning, this problem has achieved significant progress recently. The goal of this paper is to comprehensively review the deep learning-based methods for image and video inpainting. Specifically, we sort existing methods into different categories from the perspective of their high-level inpainting pipeline, present different deep learning architectures, including CNN, VAE, GAN, diffusion models, etc., and summarize techniques for module design. We review the training objectives and the common benchmark datasets. We present evaluation metrics for low-level pixel and high-level perceptional similarity, conduct a performance evaluation, and discuss the strengths and weaknesses of representative inpainting methods. We also discuss related real-world applications. Finally, we discuss open challenges and suggest potential future research directions.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 161 of its 273 references that are in the map. First posted on arXiv 7 Jan 2024.

Builds on

161 of 273 references.

References · 161

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2017Squeeze-and-Excitation Networkscited by 4,763
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  8. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2017Non-local Neural Networkscited by 2,290
  11. 2015Variational Inference with Normalizing Flowscited by 2,161
  12. 2013Describing Textures in the Wildcited by 2,110
  13. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  14. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  15. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  16. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  17. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  18. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  19. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  20. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  21. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  22. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  23. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  24. 2017Deep Image Priorcited by 1,097
  25. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  26. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  27. 2022Diffusion Models in Vision: A Surveycited by 1,008
  28. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  29. 2016Least Squares Generative Adversarial Networkscited by 1,000
  30. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  31. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  32. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  33. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  34. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  35. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  36. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  37. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  38. 2017The Perception-Distortion Tradeoffcited by 582

50 most cited of 161 All 161 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 20; 2026: 10.

44 papers.

Sort:

By area:

  1. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 200
  2. 2024A Survey on Diffusion Models for Inverse Problemscited by 173
  3. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 83
  4. 2024InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Priorcited by 55
  5. 2024BrushEdit: All-In-One Image Inpainting and Editingcited by 31
  6. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  7. 2025RORem: Training a Robust Object Remover with Human-in-the-Loopcited by 23
  8. 2025Flux Already Knows -- Activating Subject-Driven Image Generation without Trainingcited by 11
  9. 2025KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolutioncited by 11
  10. 2024Bayesian Experimental Design via Contrastive Diffusionscited by 9
  11. 2026PISCO: Precise Video Instance Insertion with Sparse Controlcited by 9
  12. 2024View-consistent Object Removal in Radiance Fieldscited by 7
32 more
  1. 2024Transformer-based Image and Video Inpainting: Current Challenges and Future Directionscited by 6
  2. 2025Geometry-Aware Diffusion Models for Multiview Scene Inpaintingcited by 6
  3. 2024Replace Anyone in Videoscited by 4
  4. 2025OutDreamer: Video Outpainting with a Diffusion Transformercited by 4
  5. 2025Vision-Language Model Guided Image Restorationcited by 4
  6. 2025High-fidelity 3D Gaussian Inpainting: preserving multi-view consistency and photorealistic detailscited by 3
  7. 2024Recursive InPainting (RIP): how much information is lost under recursive inferences?cited by 2
  8. 2024DreamMix: Decoupling Object Attributes for Enhanced Editability in Customized Image Inpaintingcited by 2
  9. 2025Advances in Artificial Intelligence: A Review for the Creative Industriescited by 2
  10. 2025Creatively Upscaling Images with Global-Regional Priorscited by 2
  11. 2025Ultra High-Resolution Image Inpainting with Patch-Based Content Consistency Adaptercited by 2
  12. 2024Fill in the ____ (a Diffusion-based Image Inpainting Pipeline)cited by 1
  13. 2024Local Lesion Generation is Effective for Capsule Endoscopy Image Data Augmentation in a Limited Data Settingcited by 1
  14. 2024PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Controlcited by 1
  15. 2025A Joint Learning Framework with Feature Reconstruction and Prediction for Incomplete Satellite Image Time Series in Agricultural Semantic Segmentationcited by 1
  16. 2025When and How to Cut Classical Concerts? A Multimodal Automated Video Editing Approachcited by 1
  17. 2026GLAD: Generative Language-Assisted Visual Tracking for Low-Semantic Templatescited by 1
  18. 2026CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removalcited by 1
  19. 2024Optimised ProPainter for Video Diminished Reality Inpaintingcited by 0
  20. 2025Time-variant Image Inpainting via Interactive Distribution Transition Estimationcited by 0
  21. 2025Mastering Regional 3DGS: Locating, Initializing, and Editing with Diverse 2D Priorscited by 0
  22. 2025Sparse Transformer for Ultra-sparse Sampled Video Compressive Sensingcited by 0
  23. 2025Inpainting the Red Planet: Diffusion Models for the Reconstruction of Martian Environments in Virtual Realitycited by 0
  24. 2025An Interdisciplinary and Cross-Task Review on Missing Data Imputationcited by 0
  25. 2025Generative deep learning for foundational video translation in ultrasoundcited by 0
  26. 2026Conditional Generative Framework with Peak-Aware Attention for Robust Chemical Detection under Interferencescited by 0
  27. 2026Transformer-Based Inpainting for Real-Time 3D Streaming in Sparse Multi-Camera Setupscited by 0
  28. 2026Quantization with Unified Adaptive Distillation to enable multi-LoRA based one-for-all Generative Vision Models on edgecited by 0
  29. 2026High-Fidelity Mural Restoration via a Unified Hybrid Mask-Aware Transformercited by 0
  30. 2026Video Reconstruction using Diffusion-based Image-to-Video Generation with Trajectory Guidancecited by 0
  31. 2026Diffusion Models in Medical Image Inpainting: Challenges, Solution Taxonomy, and Future Directionscited by 0
  32. 2026Fast Object Removal Attacks on Safety-Critical Video-based Perception Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Inst-Inpaint: Instructing to Remove Objects with Diffusion Models
  2. 2023DiffGANPaint: Fast Inpainting Using Denoising Diffusion GANs
  3. 2025Towards Seamless Borders: A Method for Mitigating Inconsistencies in Image Inpainting and Outpainting
  4. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them First
  5. 2024Semantically Consistent Video Inpainting with Conditional Diffusion Models
  6. 2025ESDiff: Encoding Strategy-inspired Diffusion Model with Few-shot Learning for Color Image Inpainting
  7. 2024Diffusion-based image inpainting with internal learning
  8. 2025VipDiff: Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models