# Panoptic-based Image Synthesis

arXiv: https://arxiv.org/abs/2004.10289 · HTML: https://mlsanity.com/paper/2004.10289/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2020.

## Built on by (most cited first, 23 of 23)

- [Diverse Semantic Image Synthesis via Probability Distribution Modeling](https://mlsanity.com/paper/2103.06878/) (2021)
- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [Dual Contrastive Loss and Attention for GANs](https://mlsanity.com/paper/2103.16748/) (2021)
- [Diverse Inpainting and Editing with GAN Inversion](https://mlsanity.com/paper/2307.15033/) (2023)
- [View Generalization for Single Image Textured 3D Models](https://mlsanity.com/paper/2106.06533/) (2021)
- [Panoptic Segmentation: A Review](https://mlsanity.com/paper/2111.10250/) (2021)
- [VecGAN: Image-to-Image Translation with Interpretable Latent Directions](https://mlsanity.com/paper/2207.03411/) (2022)
- [Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis](https://mlsanity.com/paper/2203.16898/) (2022)
- [Fine Detailed Texture Learning for 3D Meshes with Generative Models](https://mlsanity.com/paper/2203.09362/) (2022)
- [Image-to-Image Translation with Disentangled Latent Vectors for Face Editing](https://mlsanity.com/paper/2301.04628/) (2023)
- [Progressive Learning of 3D Reconstruction Network from 2D GAN Data](https://mlsanity.com/paper/2305.11102/) (2023)
- [Reference-Based 3D-Aware Image Editing with Triplanes](https://mlsanity.com/paper/2404.03632/) (2024)
- [Layout-to-Image Translation with Double Pooling Generative Adversarial Networks](https://mlsanity.com/paper/2108.12900/) (2021)
- [Benchmarking the Robustness of Instance Segmentation Models](https://mlsanity.com/paper/2109.01123/) (2021)
- [Warping the Residuals for Image Editing with StyleGAN](https://mlsanity.com/paper/2312.11422/) (2023)
- [Towards Single Stage Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2106.10309/) (2021)
- [Panoptic-aware Image-to-Image Translation](https://mlsanity.com/paper/2112.01926/) (2021)
- [CASAPose: Class-Adaptive and Semantic-Aware Multi-Object Pose Estimation](https://mlsanity.com/paper/2210.05318/) (2022)
- [Refining 3D Human Texture Estimation from a Single Image](https://mlsanity.com/paper/2303.03471/) (2023)
- [Generative Adversarial Learning via Kernel Density Discrimination](https://mlsanity.com/paper/2107.06197/) (2021)
- [Diverse Semantic Image Editing with Style Codes](https://mlsanity.com/paper/2309.13975/) (2023)
- [Style-Guided Inference of Transformer for High-resolution Image Synthesis](https://mlsanity.com/paper/2210.05533/) (2022)
- [StyleRes: Transforming the Residuals for Real Image Editing with StyleGAN](https://mlsanity.com/paper/2212.14359/) (2022)

## Abstract (arXiv)

Conditional image synthesis for generating photorealistic images serves various applications for content editing to content generation. Previous conditional image synthesis algorithms mostly rely on semantic maps, and often fail in complex environments where multiple instances occlude each other. We propose a panoptic aware image synthesis network to generate high fidelity and photorealistic images conditioned on panoptic maps which unify semantic and instance information. To achieve this, we efficiently use panoptic maps in convolution and upsampling layers. We show that with the proposed changes to the generator, we can improve on the previous state-of-the-art methods by generating images in complex instance interaction environments in higher fidelity and tiny objects in more details. Furthermore, our proposed method also outperforms the previous state-of-the-art methods in metrics of mean IoU (Intersection over Union), and detAP (Detection Average Precision).