# Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting arXiv: https://arxiv.org/abs/2403.19898 ยท HTML: https://mlsanity.com/paper/2403.19898/ ## Key facts - Cited by 40 papers in the map (data through 14 Oct 2026). - Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. - Builds on 5 of its 8 references that are in the map. - First posted on arXiv 29 Mar 2024. ## Placed in - Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/) ## Builds on (most cited first, 5 of 5) - [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022) - [Image Restoration with Mean-Reverting Stochastic Differential Equations](https://mlsanity.com/paper/2301.11699/) (2023) - [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019) - [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021) - [Diverse Image Inpainting with Bidirectional and Autoregressive Transformers](https://mlsanity.com/paper/2104.12335/) (2021) ## Built on by (most cited first, 30 of 40) - [Visual-Instructed Degradation Diffusion for All-in-One Image Restoration](https://mlsanity.com/paper/2506.16960/) (2025) - [RAD: Region-Aware Diffusion Models for Image Inpainting](https://mlsanity.com/paper/2412.09191/) (2024) - [Follow-Your-Preference: Towards Preference-Aligned Image Inpainting](https://mlsanity.com/paper/2509.23082/) (2025) - [DreamLayer: Simultaneous Multi-Layer Generation via Diffusion Mode](https://mlsanity.com/paper/2503.12838/) (2025) - [Reversing Flow for Image Restoration](https://mlsanity.com/paper/2506.16961/) (2025) - [GeoRemover: Removing Objects and Their Causal Visual Artifacts](https://mlsanity.com/paper/2509.18538/) (2025) - [MoEdit: On Learning Quantity Perception for Multi-object Image Editing](https://mlsanity.com/paper/2503.10112/) (2025) - [Manifold-based Incomplete Multi-view Clustering via Bi-Consistency Guidance](https://mlsanity.com/paper/2405.10987/) (2024) - [Few-Shot Referring Video Single- and Multi-Object Segmentation via Cross-Modal Affinity with Instance Sequence Matching](https://mlsanity.com/paper/2504.13710/) (2025) - [BVINet: Unlocking Blind Video Inpainting with Zero Annotations](https://mlsanity.com/paper/2502.01181/) (2025) - [MDE-Edit: Masked Dual-Editing for Multi-Object Image Editing via Diffusion Models](https://mlsanity.com/paper/2505.05101/) (2025) - [Towards Reliable Identification of Diffusion-based Image Manipulations](https://mlsanity.com/paper/2506.05466/) (2025) - [AsyncDSB: Schedule-Asynchronous Diffusion Schr"odinger Bridge for Image Inpainting](https://mlsanity.com/paper/2412.08149/) (2024) - [REDEditing: Relationship-Driven Precise Backdoor Poisoning on Text-to-Image Diffusion Models](https://mlsanity.com/paper/2504.14554/) (2025) - [GuidPaint: Class-Guided Image Inpainting with Diffusion Models](https://mlsanity.com/paper/2507.21627/) (2025) - [IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed](https://mlsanity.com/paper/2509.11638/) (2025) - [Semi-supervised Latent Disentangled Diffusion Model for Textile Pattern Generation](https://mlsanity.com/paper/2603.16747/) (2026) - [FCDM: A Physics-Guided Bidirectional Frequency Aware Convolution and Diffusion-Based Model for Sinogram Inpainting](https://mlsanity.com/paper/2409.06714/) (2024) - [Unsupervised Domain Adaptive Person Search via Dual Self-Calibration](https://mlsanity.com/paper/2412.16506/) (2024) - [Frequency-Domain Fusion Transformer for Image Inpainting](https://mlsanity.com/paper/2506.18437/) (2025) - [Preserve Anything: Controllable Image Synthesis with Object Preservation](https://mlsanity.com/paper/2506.22531/) (2025) - [Back to the Feature: Explaining Video Classifiers with Video Counterfactual Explanations](https://mlsanity.com/paper/2511.20295/) (2025) - [FlashClear: Ultra-Fast Image Content Removal via Efficient Step Distillation and Feature Caching](https://mlsanity.com/paper/2605.09003/) (2026) - [DORS: Dynamic Attention Routing for Diffusion-based Object Removal in Dense Scenes](https://mlsanity.com/paper/2607.16656/) (2026) - [PIDiff: Image Customization for Personalized Identities with Diffusion Models](https://mlsanity.com/paper/2505.05081/) (2025) - [Exploring Diffusion with Test-Time Training on Efficient Image Restoration](https://mlsanity.com/paper/2506.14541/) (2025) - [Security Risk of Misalignment between Text and Image in Multi-modal Model](https://mlsanity.com/paper/2510.26105/) (2025) - [KAO: Kernel-Adaptive Optimization in Diffusion for Satellite Image](https://mlsanity.com/paper/2511.02462/) (2025) - [Semantic-Guided Two-Stage GAN for Face Inpainting with Hybrid Perceptual Encoding](https://mlsanity.com/paper/2512.05039/) (2025) - [Structure-constrained Language-informed Diffusion Model for Unpaired Low-dose Computed Tomography Angiography Reconstruction](https://mlsanity.com/paper/2601.20304/) (2026) ## Abstract (arXiv) Denoising diffusion probabilistic models for image inpainting aim to add the noise to the texture of image during the forward process and recover masked regions with unmasked ones of the texture via the reverse denoising process. Despite the meaningful semantics generation, the existing arts suffer from the semantic discrepancy between masked and unmasked regions, since the semantically dense unmasked texture fails to be completely degraded while the masked regions turn to the pure noise in diffusion process, leading to the large discrepancy between them. In this paper, we aim to answer how unmasked semantics guide texture denoising process;together with how to tackle the semantic discrepancy, to facilitate the consistent and meaningful semantics generation. To this end, we propose a novel structure-guided diffusion model named StrDiffusion, to reformulate the conventional texture denoising process under structure guidance to derive a simplified denoising objective for image inpainting, while revealing: 1) the semantically sparse structure is beneficial to tackle semantic discrepancy in early stage, while dense texture generates reasonable semantics in late stage; 2) the semantics from unmasked regions essentially offer the time-dependent structure guidance for the texture denoising process, benefiting from the time-dependent sparsity of the structure semantics. For the denoising process, a structure-guided neural network is trained to estimate the simplified denoising objective by exploiting the consistency of the denoised structure between masked and unmasked regions. Besides, we devise an adaptive resampling strategy as a formal criterion as whether structure is competent to guide the texture denoising process, while regulate their semantic correlations. Extensive experiments validate the merits of StrDiffusion over the state-of-the-arts. Our code is available at https://github.com/htyjers/StrDiffusion.