Instance-Aware Image Completion
Not on the map. Why?
Abstract (arXiv)
Image completion is a task that aims to fill in the missing region of a masked image with plausible contents. However, existing image completion methods tend to fill in the missing region with the surrounding texture instead of hallucinating a visual instance that is suitable in accordance with the context of the scene. In this work, we propose a novel image completion model, dubbed ImComplete, that hallucinates the missing instance that harmonizes well with - and thus preserves - the original context. ImComplete first adopts a transformer architecture that considers the visible instances and the location of the missing region. Then, ImComplete completes the semantic segmentation masks within the missing region, providing pixel-level semantic and structural guidance. Finally, the image synthesis blocks generate photo-realistic content. We perform a comprehensive evaluation of the results in terms of visual quality (LPIPS and FID) and contextual preservation scores (CLIPscore and object detection accuracy) with COCO-panoptic and Visual Genome datasets. Experimental results show the superiority of ImComplete on various natural images.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 47 of its 60 references that are in the map. First posted on arXiv 22 Oct 2022.
Builds on
47 of 60 references.
References · 47
- 2017Attention Is All You Needcited by 59,932
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2020End-to-End Object Detection with Transformerscited by 7,384
35 more
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
- 2022Generative Adversarial Networkscited by 5,603
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
- 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
- 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
- 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
- 2022MaskGIT: Masked Generative Image Transformercited by 1,082
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
- 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
- 2018Generative Image Inpainting with Contextual Attentioncited by 578
- 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
- 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
- 2017Geometric GANcited by 232
- 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
- 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
- 2017Modulating early visual processing by languagecited by 167
- 2021EditGAN: High-Precision Semantic Image Editingcited by 156
- 2020You Only Need Adversarial Supervision for Semantic Image Synthesiscited by 129
- 2019SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Colorcited by 105
- 2019Coherent Semantic Attention for Image Inpaintingcited by 97
- 2020Bringing Old Photos Back to Lifecited by 95
- 2018Shift-Net: Image Inpainting via Deep Feature Rearrangementcited by 87
- 2018SPG-Net: Segmentation Prediction and Guidance Network for Image Inpaintingcited by 58
- 2017Contextual-based Image Inpainting: Infer, Match, and Translatecited by 50
- 2020Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenescited by 43
- 2022StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesiscited by 38
- 2018Adversarial Scene Editing: Automatic Object Removal from Weak Supervisioncited by 27
Built on by
Citing papers per year: 2023: 1.
1 paper.