# Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis

arXiv: https://arxiv.org/abs/2003.11571 · HTML: https://mlsanity.com/paper/2003.11571/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2020.

## Built on by (most cited first, 30 of 49)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)
- [Instance-Conditioned GAN](https://mlsanity.com/paper/2109.05070/) (2021)
- [TrailBlazer: Trajectory Control for Diffusion-Based Video Generation](https://mlsanity.com/paper/2401.00896/) (2023)
- [Directed Diffusion: Direct Control of Object Placement through Attention Guidance](https://mlsanity.com/paper/2302.13153/) (2023)
- [Controllable Text-to-Image Generation with GPT-4](https://mlsanity.com/paper/2305.18583/) (2023)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [Modeling Image Composition for Complex Scene Generation](https://mlsanity.com/paper/2206.00923/) (2022)
- [SceneComposer: Any-Level Semantic Image Synthesis](https://mlsanity.com/paper/2211.11742/) (2022)
- [Context-Aware Layout to Image Generation with Enhanced Object Appearance](https://mlsanity.com/paper/2103.11897/) (2021)
- [HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2410.14324/) (2024)
- [High-Resolution Complex Scene Synthesis with Transformers](https://mlsanity.com/paper/2105.06458/) (2021)
- [SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation](https://mlsanity.com/paper/2308.10156/) (2023)
- [LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.12342/) (2023)
- [LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts](https://mlsanity.com/paper/2308.06713/) (2023)
- [IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation](https://mlsanity.com/paper/2409.08240/) (2024)
- [DoodleFormer: Creative Sketch Drawing with Transformers](https://mlsanity.com/paper/2112.03258/) (2021)
- [Interactive Image Synthesis with Panoptic Layout Generation](https://mlsanity.com/paper/2203.02104/) (2022)
- [Editable Image Elements for Controllable Synthesis](https://mlsanity.com/paper/2404.16029/) (2024)
- [AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style](https://mlsanity.com/paper/2103.13722/) (2021)
- [DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Models](https://mlsanity.com/paper/2502.14779/) (2025)
- [Follow-Your-Color: Multi-Instance Sketch Colorization](https://mlsanity.com/paper/2503.16948/) (2025)
- [Layout-Bridging Text-to-Image Synthesis](https://mlsanity.com/paper/2208.06162/) (2022)
- [Automatic Controllable Colorization via Imagination](https://mlsanity.com/paper/2404.05661/) (2024)
- [Extracting Semantic Knowledge from GANs with Unsupervised Learning](https://mlsanity.com/paper/2211.16710/) (2022)
- [Controllable Image Generation via Collage Representations](https://mlsanity.com/paper/2304.13722/) (2023)
- [DT2I: Dense Text-to-Image Generation from Region Descriptions](https://mlsanity.com/paper/2204.02035/) (2022)

## Abstract (arXiv)

With the remarkable recent progress on learning deep generative models, it becomes increasingly interesting to develop models for controllable image synthesis from reconfigurable inputs. This paper focuses on a recent emerged task, layout-to-image, to learn generative models that are capable of synthesizing photo-realistic images from spatial layout (i.e., object bounding boxes configured in an image lattice) and style (i.e., structural and appearance variations encoded by latent vectors). This paper first proposes an intuitive paradigm for the task, layout-to-mask-to-image, to learn to unfold object masks of given bounding boxes in an input layout to bridge the gap between the input layout and synthesized images. Then, this paper presents a method built on Generative Adversarial Networks for the proposed layout-to-mask-to-image with style control at both image and mask levels. Object masks are learned from the input layout and iteratively refined along stages in the generator network. Style control at the image level is the same as in vanilla GANs, while style control at the object mask level is realized by a proposed novel feature normalization scheme, Instance-Sensitive and Layout-Aware Normalization. In experiments, the proposed method is tested in the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained.