# Coherent Semantic Attention for Image Inpainting

arXiv: https://arxiv.org/abs/1905.12384 · HTML: https://mlsanity.com/paper/1905.12384/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 30 of 97)

- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models](https://mlsanity.com/paper/2307.00619/) (2023)
- [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [Ambient Diffusion: Learning Clean Distributions from Corrupted Data](https://mlsanity.com/paper/2305.19256/) (2023)
- [Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE](https://mlsanity.com/paper/2103.10022/) (2021)
- [Aggregated Contextual Transformations for High-Resolution Image Inpainting](https://mlsanity.com/paper/2104.01431/) (2021)
- [PD-GAN: Probabilistic Diverse GAN for Image Inpainting](https://mlsanity.com/paper/2105.02201/) (2021)
- [Intermediate Layer Optimization for Inverse Problems using Deep Generative Models](https://mlsanity.com/paper/2102.07364/) (2021)
- [Image Inpainting via Conditional Texture and Structure Dual Generation](https://mlsanity.com/paper/2108.09760/) (2021)
- [Countering Malicious DeepFakes: Survey, Battleground, and Horizon](https://mlsanity.com/paper/2103.00218/) (2021)
- [SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting](https://mlsanity.com/paper/2109.01068/) (2021)
- [Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation](https://mlsanity.com/paper/2203.14668/) (2022)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [Reference-guided Controllable Inpainting of Neural Radiance Fields](https://mlsanity.com/paper/2304.09677/) (2023)
- [Reduce Information Loss in Transformers for Pluralistic Image Inpainting](https://mlsanity.com/paper/2205.05076/) (2022)
- [SGUIE-Net: Semantic Attention Guided Underwater Image Enhancement with Multi-Scale Perception](https://mlsanity.com/paper/2201.02832/) (2022)
- [Improving Diffusion-Based Image Synthesis with Context Prediction](https://mlsanity.com/paper/2401.02015/) (2024)
- [DeFLOCNet: Deep Image Editing via Flexible Low-level Controls](https://mlsanity.com/paper/2103.12723/) (2021)
- [BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning](https://mlsanity.com/paper/2112.06179/) (2021)
- [Learning a Sketch Tensor Space for Image Inpainting of Man-made Scenes](https://mlsanity.com/paper/2103.15087/) (2021)
- [Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand](https://mlsanity.com/paper/2208.03382/) (2022)
- [RORem: Training a Robust Object Remover with Human-in-the-Loop](https://mlsanity.com/paper/2501.00740/) (2025)
- [Improving Cross-modal Alignment for Text-Guided Image Inpainting](https://mlsanity.com/paper/2301.11362/) (2023)
- [Delving StyleGAN Inversion for Image Editing: A Foundation Latent Space Viewpoint](https://mlsanity.com/paper/2211.11448/) (2022)
- [Layout-Guided Novel View Synthesis from a Single Indoor Panorama](https://mlsanity.com/paper/2103.17022/) (2021)
- [High-Fidelity Image Inpainting with GAN Inversion](https://mlsanity.com/paper/2208.11850/) (2022)
- [Human MotionFormer: Transferring Human Motions with Vision Transformers](https://mlsanity.com/paper/2302.11306/) (2023)

## Abstract (arXiv)

The latest deep learning-based approaches have shown promising results for the challenging task of inpainting missing regions of an image. However, the existing methods often generate contents with blurry textures and distorted structures due to the discontinuity of the local pixels. From a semantic-level perspective, the local pixel discontinuity is mainly because these methods ignore the semantic relevance and feature continuity of hole regions. To handle this problem, we investigate the human behavior in repairing pictures and propose a fined deep generative model-based approach with a novel coherent semantic attention (CSA) layer, which can not only preserve contextual structure but also make more effective predictions of missing parts by modeling the semantic relevance between the holes features. The task is divided into rough, refinement as two steps and model each step with a neural network under the U-Net architecture, where the CSA layer is embedded into the encoder of refinement step. To stabilize the network training process and promote the CSA layer to learn more effective parameters, we propose a consistency loss to enforce the both the CSA layer and the corresponding layer of the CSA in decoder to be close to the VGG feature layer of a ground truth image simultaneously. The experiments on CelebA, Places2, and Paris StreetView datasets have validated the effectiveness of our proposed methods in image inpainting tasks and can obtain images with a higher quality as compared with the existing state-of-the-art approaches.