# PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph

arXiv: https://arxiv.org/abs/1905.01608 · HTML: https://mlsanity.com/paper/1905.01608/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2019.

## Built on by (most cited first, 30 of 44)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [Learning to Compose Visual Relations](https://mlsanity.com/paper/2111.09297/) (2021)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training](https://mlsanity.com/paper/2211.11138/) (2022)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis](https://mlsanity.com/paper/2304.14573/) (2023)
- [Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2112.09828/) (2021)
- [VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming](https://mlsanity.com/paper/2202.04178/) (2022)
- [SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance](https://mlsanity.com/paper/2405.15321/) (2024)
- [AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style](https://mlsanity.com/paper/2103.13722/) (2021)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation](https://mlsanity.com/paper/2410.00447/) (2024)
- [Memory-Driven Text-to-Image Generation](https://mlsanity.com/paper/2208.07022/) (2022)
- [Exploiting Relationship for Complex-scene Image Generation](https://mlsanity.com/paper/2104.00356/) (2021)
- [iPLAN: Interactive and Procedural Layout Planning](https://mlsanity.com/paper/2203.14412/) (2022)
- [Transformer-based Image Generation from Scene Graphs](https://mlsanity.com/paper/2303.04634/) (2023)
- [PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation](https://mlsanity.com/paper/2207.11790/) (2022)
- [Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-Image Generation](https://mlsanity.com/paper/2210.09549/) (2022)
- [Single-Stage Visual Relationship Learning using Conditional Queries](https://mlsanity.com/paper/2306.05689/) (2023)
- [ESCA: Contextualizing Embodied Agents via Scene-Graph Generation](https://mlsanity.com/paper/2510.15963/) (2025)
- [Controllable Image Generation via Collage Representations](https://mlsanity.com/paper/2304.13722/) (2023)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach](https://mlsanity.com/paper/2507.04815/) (2025)
- [ALR-GAN: Adaptive Layout Refinement for Text-to-Image Synthesis](https://mlsanity.com/paper/2304.06297/) (2023)
- [Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models](https://mlsanity.com/paper/2401.01130/) (2024)
- [Towards Book Cover Design via Layout Graphs](https://mlsanity.com/paper/2105.11088/) (2021)
- [SPAN: Learning Similarity between Scene Graphs and Images with Transformers](https://mlsanity.com/paper/2304.00590/) (2023)

## Abstract (arXiv)

Despite some exciting progress on high-quality image generation from structured(scene graphs) or free-form(sentences) descriptions, most of them only guarantee the image-level semantical consistency, i.e. the generated image matching the semantic meaning of the description. They still lack the investigations on synthesizing the images in a more controllable way, like finely manipulating the visual appearance of every object. Therefore, to generate the images with preferred objects and rich interactions, we propose a semi-parametric method, PasteGAN, for generating the image from the scene graph and the image crops, where spatial arrangements of the objects and their pair-wise relationships are defined by the scene graph and the object appearances are determined by the given object crops. To enhance the interactions of the objects in the output, we design a Crop Refining Network and an Object-Image Fuser to embed the objects as well as their relationships into one map. Multiple losses work collaboratively to guarantee the generated images highly respecting the crops and complying with the scene graphs while maintaining excellent image quality. A crop selector is also proposed to pick the most-compatible crops from our external object tank by encoding the interactions around the objects in the scene graph if the crops are not provided. Evaluated on Visual Genome and COCO-Stuff dataset, our proposed method significantly outperforms the SOTA methods on Inception Score, Diversity Score and Fr'echet Inception Distance. Extensive experiments also demonstrate our method's ability to generate complex and diverse images with given objects.