# Image Synthesis From Reconfigurable Layout and Style

arXiv: https://arxiv.org/abs/1908.07500 · HTML: https://mlsanity.com/paper/1908.07500/

## Key facts

- Cited by 85 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 20 Aug 2019.

## Built on by (most cited first, 30 of 85)

- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [Composer: Creative and Controllable Image Synthesis with Composable Conditions](https://mlsanity.com/paper/2302.09778/) (2023)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [Instance-Conditioned GAN](https://mlsanity.com/paper/2109.05070/) (2021)
- [LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation](https://mlsanity.com/paper/2308.05095/) (2023)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis](https://mlsanity.com/paper/2208.13753/) (2022)
- [LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation](https://mlsanity.com/paper/2302.08908/) (2023)
- [Controllable Text-to-Image Generation with GPT-4](https://mlsanity.com/paper/2305.18583/) (2023)
- [Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training](https://mlsanity.com/paper/2211.11138/) (2022)
- [GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation](https://mlsanity.com/paper/2306.04607/) (2023)
- [LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts](https://mlsanity.com/paper/2310.10640/) (2023)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis](https://mlsanity.com/paper/2108.12141/) (2021)
- [Modeling Image Composition for Complex Scene Generation](https://mlsanity.com/paper/2206.00923/) (2022)
- [SceneComposer: Any-Level Semantic Image Synthesis](https://mlsanity.com/paper/2211.11742/) (2022)
- [Context-Aware Layout to Image Generation with Enhanced Object Appearance](https://mlsanity.com/paper/2103.11897/) (2021)
- [Unbalanced Feature Transport for Exemplar-based Image Translation](https://mlsanity.com/paper/2106.10482/) (2021)
- [SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis](https://mlsanity.com/paper/2304.14573/) (2023)
- [DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception](https://mlsanity.com/paper/2403.13304/) (2024)
- [AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generation](https://mlsanity.com/paper/2411.15497/) (2024)
- [High-Resolution Complex Scene Synthesis with Transformers](https://mlsanity.com/paper/2105.06458/) (2021)
- [SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation](https://mlsanity.com/paper/2308.10156/) (2023)
- [Localized Text-to-Image Generation for Free via Cross Attention Control](https://mlsanity.com/paper/2306.14636/) (2023)
- [Bi-level Feature Alignment for Versatile Image Translation and Manipulation](https://mlsanity.com/paper/2107.03021/) (2021)
- [LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts](https://mlsanity.com/paper/2308.06713/) (2023)

## Abstract (arXiv)

Despite remarkable recent progress on both unconditional and conditional image synthesis, it remains a long-standing problem to learn generative models that are capable of synthesizing realistic and sharp images from reconfigurable spatial layout (i.e., bounding boxes + class labels in an image lattice) and style (i.e., structural and appearance variations encoded by latent vectors), especially at high resolution. By reconfigurable, it means that a model can preserve the intrinsic one-to-many mapping from a given layout to multiple plausible images with different styles, and is adaptive with respect to perturbations of a layout and style latent code. In this paper, we present a layout- and style-based architecture for generative adversarial networks (termed LostGANs) that can be trained end-to-end to generate images from reconfigurable layout and style. Inspired by the vanilla StyleGAN, the proposed LostGAN consists of two new components: (i) learning fine-grained mask maps in a weakly-supervised manner to bridge the gap between layouts and images, and (ii) learning object instance-specific layout-aware feature normalization (ISLA-Norm) in the generator to realize multi-object style generation. In experiments, the proposed method is tested on the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained. The code and pretrained models are available at urlhttps://github.com/iVMCL/LostGANs.