# Learning to Incorporate Structure Knowledge for Image Inpainting

arXiv: https://arxiv.org/abs/2002.04170 · HTML: https://mlsanity.com/paper/2002.04170/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2020.

## Built on by (most cited first, 30 of 37)

- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding](https://mlsanity.com/paper/2203.00867/) (2022)
- [Image Inpainting via Conditional Texture and Structure Dual Generation](https://mlsanity.com/paper/2108.09760/) (2021)
- [CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training](https://mlsanity.com/paper/2203.11947/) (2022)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimation](https://mlsanity.com/paper/2304.07803/) (2023)
- [SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches](https://mlsanity.com/paper/2111.15078/) (2021)
- [Learning a Sketch Tensor Space for Image Inpainting of Man-made Scenes](https://mlsanity.com/paper/2103.15087/) (2021)
- [ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors](https://mlsanity.com/paper/2210.05950/) (2022)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [Gradpaint: Gradient-Guided Inpainting with Diffusion Models](https://mlsanity.com/paper/2309.09614/) (2023)
- [Learning Prior Feature and Attention Enhanced Image Inpainting](https://mlsanity.com/paper/2208.01837/) (2022)
- [Delving Globally into Texture and Structure for Image Inpainting](https://mlsanity.com/paper/2209.08217/) (2022)
- [Visual Anomaly Detection for Images: A Survey](https://mlsanity.com/paper/2109.13157/) (2021)
- [Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion Interpolation](https://mlsanity.com/paper/2303.14926/) (2023)
- [Unbiased Multi-Modality Guidance for Image Inpainting](https://mlsanity.com/paper/2208.11844/) (2022)
- [Markpainting: Adversarial Machine Learning meets Inpainting](https://mlsanity.com/paper/2106.00660/) (2021)
- [Shape-guided Object Inpainting](https://mlsanity.com/paper/2204.07845/) (2022)
- [Generative Memory-Guided Semantic Reasoning Model for Image Inpainting](https://mlsanity.com/paper/2110.00261/) (2021)
- [Layout-guided Indoor Panorama Inpainting with Plane-aware Normalization](https://mlsanity.com/paper/2301.05624/) (2023)
- [TransRef: Multi-Scale Reference Embedding Transformer for Reference-Guided Image Inpainting](https://mlsanity.com/paper/2306.11528/) (2023)
- [Hierarchical Contrastive Learning for Pattern-Generalizable Image Corruption Detection](https://mlsanity.com/paper/2308.14061/) (2023)
- [FaceRefiner: High-Fidelity Facial Texture Refinement with Differentiable Rendering-based Style Transfer](https://mlsanity.com/paper/2601.04520/) (2026)
- [Multi-feature Co-learning for Image Inpainting](https://mlsanity.com/paper/2205.10578/) (2022)
- [Structure-Guided Image Completion with Image-level and Object-level Semantic Discriminators](https://mlsanity.com/paper/2212.06310/) (2022)
- [Attack Deterministic Conditional Image Generative Models for Diverse and Controllable Generation](https://mlsanity.com/paper/2403.08294/) (2024)
- [My Body My Choice: Human-Centric Full-Body Anonymization](https://mlsanity.com/paper/2406.09553/) (2024)
- [Continuation of Famous Art with AI: A Conditional Adversarial Network Inpainting Approach](https://mlsanity.com/paper/2110.09170/) (2021)
- [Photorealistic Facial Wrinkles Removal](https://mlsanity.com/paper/2211.01930/) (2022)
- [Learning Visual Planning Models from Partially Observed Images](https://mlsanity.com/paper/2211.15666/) (2022)

## Abstract (arXiv)

This paper develops a multi-task learning framework that attempts to incorporate the image structure knowledge to assist image inpainting, which is not well explored in previous works. The primary idea is to train a shared generator to simultaneously complete the corrupted image and corresponding structures --- edge and gradient, thus implicitly encouraging the generator to exploit relevant structure knowledge while inpainting. In the meantime, we also introduce a structure embedding scheme to explicitly embed the learned structure features into the inpainting process, thus to provide possible preconditions for image completion. Specifically, a novel pyramid structure loss is proposed to supervise structure learning and embedding. Moreover, an attention mechanism is developed to further exploit the recurrent structures and patterns in the image to refine the generated structures and contents. Through multi-task learning, structure embedding besides with attention, our framework takes advantage of the structure knowledge and outperforms several state-of-the-art methods on benchmark datasets quantitatively and qualitatively.