# Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models

arXiv: https://arxiv.org/abs/2304.03322 · HTML: https://mlsanity.com/paper/2304.03322/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 45 of its 57 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 45)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Improving Diffusion Models for Inverse Problems using Manifold Constraints](https://mlsanity.com/paper/2206.00941/) (2022)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction](https://mlsanity.com/paper/2112.05146/) (2021)
- [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://mlsanity.com/paper/2208.09392/) (2022)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Deblurring via Stochastic Refinement](https://mlsanity.com/paper/2112.02475/) (2021)
- [Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem](https://mlsanity.com/paper/2206.04119/) (2022)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)

## Built on by (most cited first, 30 of 61)

- [Practical and Asymptotically Exact Conditional Sampling in Diffusion Models](https://mlsanity.com/paper/2306.17775/) (2023)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion](https://mlsanity.com/paper/2403.06976/) (2024)
- [Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models](https://mlsanity.com/paper/2503.01774/) (2025)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [Diffusion Models in Low-Level Vision: A Survey](https://mlsanity.com/paper/2406.11138/) (2024)
- [Monte Carlo guided Diffusion for Bayesian linear inverse problems](https://mlsanity.com/paper/2308.07983/) (2023)
- [Image Restoration Through Generalized Ornstein-Uhlenbeck Bridge](https://mlsanity.com/paper/2312.10299/) (2023)
- [Diffusion Facial Forgery Detection](https://mlsanity.com/paper/2401.15859/) (2024)
- [Variational Diffusion Posterior Sampling with Midpoint Guidance](https://mlsanity.com/paper/2410.09945/) (2024)
- [BrushEdit: All-In-One Image Inpainting and Editing](https://mlsanity.com/paper/2412.10316/) (2024)
- [Divide-and-Conquer Posterior Sampling for Denoising Diffusion Priors](https://mlsanity.com/paper/2403.11407/) (2024)
- [NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging](https://mlsanity.com/paper/2403.03485/) (2024)
- [Image Inpainting via Tractable Steering of Diffusion Models](https://mlsanity.com/paper/2401.03349/) (2023)
- [An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations](https://mlsanity.com/paper/2407.01014/) (2024)
- [Correcting Diffusion Generation through Resampling](https://mlsanity.com/paper/2312.06038/) (2023)
- [Neural Conditional Probability for Uncertainty Quantification](https://mlsanity.com/paper/2407.01171/) (2024)
- [Taming Diffusion Models for Image Restoration: A Review](https://mlsanity.com/paper/2409.10353/) (2024)
- [LanPaint: Training-Free Diffusion Inpainting with Asymptotically Exact and Fast Conditional Sampling](https://mlsanity.com/paper/2502.03491/) (2025)
- [Diff-Shadow: Global-guided Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2407.16214/) (2024)
- [DiffVax: Optimization-Free Image Immunization Against Diffusion-Based Editing](https://mlsanity.com/paper/2411.17957/) (2024)
- [Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization](https://mlsanity.com/paper/2401.16762/) (2024)
- [Text Image Inpainting via Global Structure-Guided Diffusion Models](https://mlsanity.com/paper/2401.14832/) (2024)
- [Semantically Consistent Video Inpainting with Conditional Diffusion Models](https://mlsanity.com/paper/2405.00251/) (2024)
- [AnyTrans: Translate AnyText in the Image with Large Scale Models](https://mlsanity.com/paper/2406.11432/) (2024)
- [Generative assimilation and prediction for weather and climate](https://mlsanity.com/paper/2503.03038/) (2025)
- [Inpaint360GS: Efficient Object-Aware 3D Inpainting via Gaussian Splatting for 360deg Scenes](https://mlsanity.com/paper/2511.06457/) (2025)
- [Timbre transfer using image-to-image denoising diffusion implicit models](https://mlsanity.com/paper/2307.04586/) (2023)
- [Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models](https://mlsanity.com/paper/2505.20789/) (2025)

## Abstract (arXiv)

Image inpainting refers to the task of generating a complete, natural image based on a partially revealed reference image. Recently, many research interests have been focused on addressing this problem using fixed diffusion models. These approaches typically directly replace the revealed region of the intermediate or final generated images with that of the reference image or its variants. However, since the unrevealed regions are not directly modified to match the context, it results in incoherence between revealed and unrevealed regions. To address the incoherence problem, a small number of methods introduce a rigorous Bayesian framework, but they tend to introduce mismatches between the generated and the reference images due to the approximation errors in computing the posterior distributions. In this paper, we propose COPAINT, which can coherently inpaint the whole image without introducing mismatches. COPAINT also uses the Bayesian framework to jointly modify both revealed and unrevealed regions, but approximates the posterior distribution in a way that allows the errors to gradually drop to zero throughout the denoising steps, thus strongly penalizing any mismatches with the reference image. Our experiments verify that COPAINT can outperform the existing diffusion-based methods under both objective and subjective metrics. The codes are available at https://github.com/UCSB-NLP-Chang/CoPaint/.