Inferring and Leveraging Parts from Object Shape for Improving Semantic Image Synthesis
Not on the map. Why?
Abstract (arXiv)
Despite the progress in semantic image synthesis, it remains a challenging problem to generate photo-realistic parts from input semantic map. Integrating part segmentation map can undoubtedly benefit image synthesis, but is bothersome and inconvenient to be provided by users. To improve part synthesis, this paper presents to infer Parts from Object ShapE (iPOSE) and leverage it for improving semantic image synthesis. However, albeit several part segmentation datasets are available, part annotations are still not provided for many object categories in semantic image synthesis. To circumvent it, we resort to few-shot regime to learn a PartNet for predicting the object part map with the guidance of pre-defined support part maps. PartNet can be readily generalized to handle a new object category when a small number (e.g., 3) of support part maps for this category are provided. Furthermore, part semantic modulation is presented to incorporate both inferred part map and semantic map for image synthesis. Experiments show that our iPOSE not only generates objects with rich part details, but also enables to control the image synthesis flexibly. And our iPOSE performs favorably against the state-of-the-art methods in terms of quantitative and qualitative evaluation. Our code will be publicly available at https://github.com/csyxwei/iPOSE.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 50 of its 61 references that are in the map. First posted on arXiv 31 May 2023.
Builds on
50 of 61 references.
References · 50
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
- 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
- 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
- 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
38 more
- 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
- 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
- 2019PANet: Few-Shot Image Semantic Segmentation with Prototype Alignmentcited by 449
- 2017One-Shot Learning for Semantic Segmentationcited by 337
- 2017Dilated Residual Networkscited by 294
- 2020Prior Guided Feature Enrichment Network for Few-Shot Segmentationcited by 271
- 2019SEAN: Image Synthesis with Semantic Region-Adaptive Normalizationcited by 253
- 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 232
- 2017Photographic Image Synthesis with Cascaded Refinement Networkscited by 223
- 2019CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learningcited by 217
- 2021Adaptive Prototype Learning and Allocation for Few-Shot Segmentationcited by 207
- 2018SG-One: Similarity Guidance Network for One-Shot Semantic Segmentationcited by 169
- 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
- 2018Instance-level Human Parsing via Part Grouping Networkcited by 134
- 2022Semantic Image Synthesis via Diffusion Modelscited by 132
- 2020You Only Need Adversarial Supervision for Semantic Image Synthesiscited by 129
- 2019Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesiscited by 108
- 2019Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generationcited by 75
- 2022Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentationcited by 65
- 2018Devil in the Details: Towards Accurate Single and Multiple Human Parsingcited by 64
- 2018Diverse Image Synthesis from Semantic Layouts via Conditional IMLEcited by 60
- 2020Generalized Few-shot Semantic Segmentationcited by 59
- 2022Learning Non-target Knowledge for Few-shot Semantic Segmentationcited by 57
- 2021Part-aware Panoptic Segmentationcited by 56
- 2021Learning Meta-class Memory for Few-Shot Semantic Segmentationcited by 56
- 2018Semi-parametric Image Synthesiscited by 50
- 2021Image Synthesis via Semantic Compositioncited by 45
- 2020SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objectscited by 43
- 2022Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentationcited by 38
- 2019Face Parsing with RoI Tanh-Warpingcited by 33
- 2015Joint Object and Part Segmentation using Deep Learned Potentialscited by 23
- 2022Towards Diverse and Faithful One-shot Adaption of Generative Adversarial Networkscited by 23
- 2017Holistic, Instance-Level Human Parsingcited by 19
- 2022Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesiscited by 16
- 2022Retrieval-based Spatially Adaptive Normalization for Semantic Image Synthesiscited by 15
- 2018Cross-domain Human Parsing via Adversarial Feature and Label Adaptationcited by 11
- 2022Dual Pyramid Generative Adversarial Networks for Semantic Image Synthesiscited by 7
- 2022Few-shot semantic segmentation via mask aggregationcited by 2
Built on by
Citing papers per year: 2024: 2.
2 papers.