# Image Generation from Scene Graphs

arXiv: https://arxiv.org/abs/1804.01622 · HTML: https://mlsanity.com/paper/1804.01622/

## Key facts

- Cited by 324 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2018.

## Built on by (most cited first, 30 of 324)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.13655/) (2023)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [Compositional Chain-of-Thought Prompting for Large Multimodal Models](https://mlsanity.com/paper/2311.17076/) (2023)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [Equivariant Subgraph Aggregation Networks](https://mlsanity.com/paper/2110.02910/) (2021)
- [Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition](https://mlsanity.com/paper/2501.03230/) (2024)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Instance-Conditioned GAN](https://mlsanity.com/paper/2109.05070/) (2021)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2103.16002/) (2021)
- [InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2402.04717/) (2024)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Paint by Word](https://mlsanity.com/paper/2103.10951/) (2021)
- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [Street-View Image Generation from a Bird's-Eye View Layout](https://mlsanity.com/paper/2301.04634/) (2023)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [CanvasVAE: Learning to Generate Vector Graphic Documents](https://mlsanity.com/paper/2108.01249/) (2021)
- [LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation](https://mlsanity.com/paper/2308.05095/) (2023)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs](https://mlsanity.com/paper/2108.08841/) (2021)
- [Multi-LoRA Composition for Image Generation](https://mlsanity.com/paper/2402.16843/) (2024)

## Abstract (arXiv)

To truly understand the visual world our models should be able not only to recognize images but also generate them. To this end, there has been exciting recent progress on generating images from natural language descriptions. These methods give stunning results on limited domains such as descriptions of birds or flowers, but struggle to faithfully reproduce complex sentences with many objects and relationships. To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships. Our model uses graph convolution to process input graphs, computes a scene layout by predicting bounding boxes and segmentation masks for objects, and converts the layout to an image with a cascaded refinement network. The network is trained adversarially against a pair of discriminators to ensure realistic outputs. We validate our approach on Visual Genome and COCO-Stuff, where qualitative results, ablations, and user studies demonstrate our method's ability to generate complex images with multiple objects.