# Interactive Image Generation Using Scene Graphs

arXiv: https://arxiv.org/abs/1905.03743 · HTML: https://mlsanity.com/paper/1905.03743/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2019.

## Built on by (most cited first, 17 of 17)

- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Learning to Compose Visual Relations](https://mlsanity.com/paper/2111.09297/) (2021)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis](https://mlsanity.com/paper/2304.14573/) (2023)
- [VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming](https://mlsanity.com/paper/2202.04178/) (2022)
- [Chatting Makes Perfect: Chat-based Image Retrieval](https://mlsanity.com/paper/2305.20062/) (2023)
- [Transformer-based Image Generation from Scene Graphs](https://mlsanity.com/paper/2303.04634/) (2023)
- [Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-Image Generation](https://mlsanity.com/paper/2210.09549/) (2022)
- [VeriGraph: Scene Graphs for Execution Verifiable Robot Planning](https://mlsanity.com/paper/2411.10446/) (2024)
- [CLID: Controlled-Length Image Descriptions with Limited Data](https://mlsanity.com/paper/2211.14835/) (2022)
- [PRISM: Progressive Restoration for Scene Graph-based Image Manipulation](https://mlsanity.com/paper/2311.02247/) (2023)
- [Generating by Understanding: Neural Visual Generation with Logical Symbol Groundings](https://mlsanity.com/paper/2310.17451/) (2023)
- [Graph2Pix: A Graph-Based Image to Image Translation Framework](https://mlsanity.com/paper/2108.09752/) (2021)
- [Transforming Image Generation from Scene Graphs](https://mlsanity.com/paper/2207.00545/) (2022)
- [Multi-Task Edge Prediction in Temporally-Dynamic Video Graphs](https://mlsanity.com/paper/2212.02875/) (2022)
- [Targeted Therapy in Data Removal: Object Unlearning Based on Scene Graphs](https://mlsanity.com/paper/2412.00067/) (2024)

## Abstract (arXiv)

Recent years have witnessed some exciting developments in the domain of generating images from scene-based text descriptions. These approaches have primarily focused on generating images from a static text description and are limited to generating images in a single pass. They are unable to generate an image interactively based on an incrementally additive text description (something that is more intuitive and similar to the way we describe an image). We propose a method to generate an image incrementally based on a sequence of graphs of scene descriptions (scene-graphs). We propose a recurrent network architecture that preserves the image content generated in previous steps and modifies the cumulative image as per the newly provided scene information. Our model utilizes Graph Convolutional Networks (GCN) to cater to variable-sized scene graphs along with Generative Adversarial image translation networks to generate realistic multi-object images without needing any intermediate supervision during training. We experiment with Coco-Stuff dataset which has multi-object images along with annotations describing the visual scene and show that our model significantly outperforms other approaches on the same dataset in generating visually consistent images for incrementally growing scene graphs.