# SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

arXiv: https://arxiv.org/abs/1805.03356 · HTML: https://mlsanity.com/paper/1805.03356/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2018.

## Built on by (most cited first, 30 of 58)

- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE](https://mlsanity.com/paper/2103.10022/) (2021)
- [Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding](https://mlsanity.com/paper/2203.00867/) (2022)
- [Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2304.03322/) (2023)
- [CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training](https://mlsanity.com/paper/2203.11947/) (2022)
- [TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations](https://mlsanity.com/paper/2103.15982/) (2021)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [Bridging Global Context Interactions for High-Fidelity Image Completion](https://mlsanity.com/paper/2104.00845/) (2021)
- [Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness](https://mlsanity.com/paper/2104.13743/) (2021)
- [Conditional Image Generation by Conditioning Variational Auto-Encoders](https://mlsanity.com/paper/2102.12037/) (2021)
- [Learning a Sketch Tensor Space for Image Inpainting of Man-made Scenes](https://mlsanity.com/paper/2103.15087/) (2021)
- [Image Inpainting with External-internal Learning and Monochromic Bottleneck](https://mlsanity.com/paper/2104.09068/) (2021)
- [ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors](https://mlsanity.com/paper/2210.05950/) (2022)
- [Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos](https://mlsanity.com/paper/2305.16301/) (2023)
- [Paste, Inpaint and Harmonize via Denoising: Subject-Driven Image Editing with Pre-Trained Diffusion Model](https://mlsanity.com/paper/2306.07596/) (2023)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [Learning Prior Feature and Attention Enhanced Image Inpainting](https://mlsanity.com/paper/2208.01837/) (2022)
- [Certified Defences Against Adversarial Patch Attacks on Semantic Segmentation](https://mlsanity.com/paper/2209.05980/) (2022)
- [Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration](https://mlsanity.com/paper/2306.06513/) (2023)
- [Facial-Sketch Synthesis: A New Challenge](https://mlsanity.com/paper/2112.15439/) (2021)
- [MapEx: Indoor Structure Exploration with Probabilistic Information Gain from Global Map Predictions](https://mlsanity.com/paper/2409.15590/) (2024)
- [Unbiased Multi-Modality Guidance for Image Inpainting](https://mlsanity.com/paper/2208.11844/) (2022)
- [GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding](https://mlsanity.com/paper/2201.08131/) (2022)
- [Image Inpainting via Iteratively Decoupled Probabilistic Modeling](https://mlsanity.com/paper/2212.02963/) (2022)
- [EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observations](https://mlsanity.com/paper/2506.17896/) (2025)
- [SemIE: Semantically-aware Image Extrapolation](https://mlsanity.com/paper/2108.13702/) (2021)
- [Deep Face Video Inpainting via UV Mapping](https://mlsanity.com/paper/2109.00681/) (2021)
- [High-Quality Pluralistic Image Completion via Code Shared VQGAN](https://mlsanity.com/paper/2204.01931/) (2022)
- [TransRef: Multi-Scale Reference Embedding Transformer for Reference-Guided Image Inpainting](https://mlsanity.com/paper/2306.11528/) (2023)

## Abstract (arXiv)

In this paper, we focus on image inpainting task, aiming at recovering the missing area of an incomplete image given the context information. Recent development in deep generative models enables an efficient end-to-end framework for image synthesis and inpainting tasks, but existing methods based on generative models don't exploit the segmentation information to constrain the object shapes, which usually lead to blurry results on the boundary. To tackle this problem, we propose to introduce the semantic segmentation information, which disentangles the inter-class difference and intra-class variation for image inpainting. This leads to much clearer recovered boundary between semantically different regions and better texture within semantically consistent segments. Our model factorizes the image inpainting process into segmentation prediction (SP-Net) and segmentation guidance (SG-Net) as two steps, which predict the segmentation labels in the missing area first, and then generate segmentation guided inpainting results. Experiments on multiple public datasets show that our approach outperforms existing methods in optimizing the image inpainting quality, and the interactive segmentation guidance provides possibilities for multi-modal predictions of image inpainting.