# Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting

arXiv: https://arxiv.org/abs/1904.07475 · HTML: https://mlsanity.com/paper/1904.07475/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2019.

## Built on by (most cited first, 30 of 94)

- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](https://mlsanity.com/paper/2312.03594/) (2023)
- [Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE](https://mlsanity.com/paper/2103.10022/) (2021)
- [Aggregated Contextual Transformations for High-Resolution Image Inpainting](https://mlsanity.com/paper/2104.01431/) (2021)
- [Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding](https://mlsanity.com/paper/2203.00867/) (2022)
- [ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion](https://mlsanity.com/paper/2403.18818/) (2024)
- [Removing Objects From Neural Radiance Fields](https://mlsanity.com/paper/2212.11966/) (2022)
- [Countering Malicious DeepFakes: Survey, Battleground, and Horizon](https://mlsanity.com/paper/2103.00218/) (2021)
- [WaveFill: A Wavelet-based Generation Network for Image Inpainting](https://mlsanity.com/paper/2107.11027/) (2021)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment](https://mlsanity.com/paper/2108.07948/) (2021)
- [Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness](https://mlsanity.com/paper/2104.13743/) (2021)
- [UMC: A Unified Bandwidth-efficient and Multi-resolution based Collaborative Perception Framework](https://mlsanity.com/paper/2303.12400/) (2023)
- [Learning a Sketch Tensor Space for Image Inpainting of Man-made Scenes](https://mlsanity.com/paper/2103.15087/) (2021)
- [RORem: Training a Robust Object Remover with Human-in-the-Loop](https://mlsanity.com/paper/2501.00740/) (2025)
- [Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers](https://mlsanity.com/paper/2111.03481/) (2021)
- [Magic Insert: Style-Aware Drag-and-Drop](https://mlsanity.com/paper/2407.02489/) (2024)
- [Layout-Guided Novel View Synthesis from a Single Indoor Panorama](https://mlsanity.com/paper/2103.17022/) (2021)
- [TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting](https://mlsanity.com/paper/2504.00996/) (2025)
- [Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration](https://mlsanity.com/paper/2306.06513/) (2023)
- [Image-to-Video Generation via 3D Facial Dynamics](https://mlsanity.com/paper/2105.14678/) (2021)
- [PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes](https://mlsanity.com/paper/2106.00446/) (2021)
- [Delving Globally into Texture and Structure for Image Inpainting](https://mlsanity.com/paper/2209.08217/) (2022)
- [DeViT: Deformed Vision Transformers in Video Inpainting](https://mlsanity.com/paper/2209.13925/) (2022)
- [Precise Object and Effect Removal with Adaptive Target-Aware Attention](https://mlsanity.com/paper/2505.22636/) (2025)
- [Bridging the Visual Gap: Wide-Range Image Blending](https://mlsanity.com/paper/2103.15149/) (2021)

## Abstract (arXiv)

High-quality image inpainting requires filling missing regions in a damaged image with plausible content. Existing works either fill the regions by copying image patches or generating semantically-coherent patches from region context, while neglect the fact that both visual and semantic plausibility are highly-demanded. In this paper, we propose a Pyramid-context ENcoder Network (PEN-Net) for image inpainting by deep generative models. The PEN-Net is built upon a U-Net structure, which can restore an image by encoding contextual semantics from full resolution input, and decoding the learned semantic features back into images. Specifically, we propose a pyramid-context encoder, which progressively learns region affinity by attention from a high-level semantic feature map and transfers the learned attention to the previous low-level feature map. As the missing content can be filled by attention transfer from deep to shallow in a pyramid fashion, both visual and semantic coherence for image inpainting can be ensured. We further propose a multi-scale decoder with deeply-supervised pyramid losses and an adversarial loss. Such a design not only results in fast convergence in training, but more realistic results in testing. Extensive experiments on various datasets show the superior performance of the proposed network