# High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis

arXiv: https://arxiv.org/abs/1611.09969 · HTML: https://mlsanity.com/paper/1611.09969/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2016.

## Built on by (most cited first, 30 of 90)

- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging](https://mlsanity.com/paper/2105.14021/) (2021)
- [Aggregated Contextual Transformations for High-Resolution Image Inpainting](https://mlsanity.com/paper/2104.01431/) (2021)
- [PixelSynth: Generating a 3D-Consistent Experience from a Single Image](https://mlsanity.com/paper/2108.05892/) (2021)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [Image Inpainting via Conditional Texture and Structure Dual Generation](https://mlsanity.com/paper/2108.09760/) (2021)
- [Putting People in Their Place: Affordance-Aware Human Insertion into Scenes](https://mlsanity.com/paper/2304.14406/) (2023)
- [CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training](https://mlsanity.com/paper/2203.11947/) (2022)
- [TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations](https://mlsanity.com/paper/2103.15982/) (2021)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [FWD: Real-time Novel View Synthesis with Forward Warping and Depth](https://mlsanity.com/paper/2206.08355/) (2022)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness](https://mlsanity.com/paper/2104.13743/) (2021)
- [GeoDiffuser: Geometry-Based Image Editing with Diffusion Models](https://mlsanity.com/paper/2404.14403/) (2024)
- [Toward Accurate and Realistic Outfits Visualization with Attention to Details](https://mlsanity.com/paper/2106.06593/) (2021)
- [SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches](https://mlsanity.com/paper/2111.15078/) (2021)
- [Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoireing](https://mlsanity.com/paper/2207.09935/) (2022)
- [Online Multi-Granularity Distillation for GAN Compression](https://mlsanity.com/paper/2108.06908/) (2021)
- [Visiting the Invisible: Layer-by-Layer Completed Scene Decomposition](https://mlsanity.com/paper/2104.05367/) (2021)
- [RORem: Training a Robust Object Remover with Human-in-the-Loop](https://mlsanity.com/paper/2501.00740/) (2025)
- [LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization](https://mlsanity.com/paper/2503.08619/) (2025)
- [MPIB: An MPI-Based Bokeh Rendering Framework for Realistic Partial Occlusion Effects](https://mlsanity.com/paper/2207.08403/) (2022)
- [Generative Adversarial Networks: A Survey Towards Private and Secure Applications](https://mlsanity.com/paper/2106.03785/) (2021)
- [Cylin-Painting: Seamless 360textdegree Panoramic Image Outpainting and Beyond](https://mlsanity.com/paper/2204.08563/) (2022)
- [RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection](https://mlsanity.com/paper/2302.12464/) (2023)
- [Improving Text-guided Object Inpainting with Semantic Pre-inpainting](https://mlsanity.com/paper/2409.08260/) (2024)
- [Generalised Image Outpainting with U-Transformer](https://mlsanity.com/paper/2201.11403/) (2022)

## Abstract (arXiv)

Recent advances in deep learning have shown exciting promise in filling large holes in natural images with semantically plausible and context aware details, impacting fundamental image manipulation tasks such as object removal. While these learning-based methods are significantly more effective in capturing high-level features than prior techniques, they can only handle very low-resolution inputs due to memory limitations and difficulty in training. Even for slightly larger images, the inpainted regions would appear blurry and unpleasant boundaries become visible. We propose a multi-scale neural patch synthesis approach based on joint optimization of image content and texture constraints, which not only preserves contextual structures but also produces high-frequency details by matching and adapting patches with the most similar mid-layer feature correlations of a deep classification network. We evaluate our method on the ImageNet and Paris Streetview datasets and achieved state-of-the-art inpainting accuracy. We show our approach produces sharper and more coherent results than prior methods, especially for high-resolution images.