# Shift-Net: Image Inpainting via Deep Feature Rearrangement

arXiv: https://arxiv.org/abs/1801.09392 · HTML: https://mlsanity.com/paper/1801.09392/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2018.

## Built on by (most cited first, 30 of 87)

- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022)
- [Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE](https://mlsanity.com/paper/2103.10022/) (2021)
- [Aggregated Contextual Transformations for High-Resolution Image Inpainting](https://mlsanity.com/paper/2104.01431/) (2021)
- [Diverse Image Inpainting with Bidirectional and Autoregressive Transformers](https://mlsanity.com/paper/2104.12335/) (2021)
- [PD-GAN: Probabilistic Diverse GAN for Image Inpainting](https://mlsanity.com/paper/2105.02201/) (2021)
- [Defect-GAN: High-Fidelity Defect Synthesis for Automated Defect Inspection](https://mlsanity.com/paper/2103.15158/) (2021)
- [Image Inpainting via Conditional Texture and Structure Dual Generation](https://mlsanity.com/paper/2108.09760/) (2021)
- [Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance](https://mlsanity.com/paper/2412.12974/) (2024)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [Bridging Global Context Interactions for High-Fidelity Image Completion](https://mlsanity.com/paper/2104.00845/) (2021)
- [Reference-guided Controllable Inpainting of Neural Radiance Fields](https://mlsanity.com/paper/2304.09677/) (2023)
- [Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness](https://mlsanity.com/paper/2104.13743/) (2021)
- [MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting](https://mlsanity.com/paper/2203.06304/) (2022)
- [DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models](https://mlsanity.com/paper/2407.01519/) (2024)
- [Pixel-Inconsistency Modeling for Image Manipulation Localization](https://mlsanity.com/paper/2310.00234/) (2023)
- [DeFLOCNet: Deep Image Editing via Flexible Low-level Controls](https://mlsanity.com/paper/2103.12723/) (2021)
- [RORem: Training a Robust Object Remover with Human-in-the-Loop](https://mlsanity.com/paper/2501.00740/) (2025)
- [Layout-Guided Novel View Synthesis from a Single Indoor Panorama](https://mlsanity.com/paper/2103.17022/) (2021)
- [Detailed Avatar Recovery from Single Image](https://mlsanity.com/paper/2108.02931/) (2021)
- [High-Fidelity Image Inpainting with GAN Inversion](https://mlsanity.com/paper/2208.11850/) (2022)
- [RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection](https://mlsanity.com/paper/2302.12464/) (2023)
- [Reference-Guided Large-Scale Face Inpainting with Identity and Texture Control](https://mlsanity.com/paper/2303.07014/) (2023)
- [Enhancement of Novel View Synthesis Using Omnidirectional Image Completion](https://mlsanity.com/paper/2203.09957/) (2022)
- [Image Inpainting via Iteratively Decoupled Probabilistic Modeling](https://mlsanity.com/paper/2212.02963/) (2022)
- [Deep Face Video Inpainting via UV Mapping](https://mlsanity.com/paper/2109.00681/) (2021)
- [Generative Memory-Guided Semantic Reasoning Model for Image Inpainting](https://mlsanity.com/paper/2110.00261/) (2021)

## Abstract (arXiv)

Deep convolutional networks (CNNs) have exhibited their potential in image inpainting for producing plausible results. However, in most existing methods, e.g., context encoder, the missing parts are predicted by propagating the surrounding convolutional features through a fully connected layer, which intends to produce semantically plausible but blurry result. In this paper, we introduce a special shift-connection layer to the U-Net architecture, namely Shift-Net, for filling in missing regions of any shape with sharp structures and fine-detailed textures. To this end, the encoder feature of the known region is shifted to serve as an estimation of the missing parts. A guidance loss is introduced on decoder feature to minimize the distance between the decoder feature after fully connected layer and the ground-truth encoder feature of the missing parts. With such constraint, the decoder feature in missing region can be used to guide the shift of encoder feature in known region. An end-to-end learning algorithm is further developed to train the Shift-Net. Experiments on the Paris StreetView and Places datasets demonstrate the efficiency and effectiveness of our Shift-Net in producing sharper, fine-detailed, and visually plausible results. The codes and pre-trained models are available at https://github.com/Zhaoyi-Yan/Shift-Net.