# High-Resolution Complex Scene Synthesis with Transformers

arXiv: https://arxiv.org/abs/2105.06458 · HTML: https://mlsanity.com/paper/2105.06458/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 13 May 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [Image Synthesis From Reconfigurable Layout and Style](https://mlsanity.com/paper/1908.07500/) (2019)
- [Specifying Object Attributes and Relations in Interactive Scene Generation](https://mlsanity.com/paper/1909.05379/) (2019)
- [Object-Centric Image Generation from Layouts](https://mlsanity.com/paper/2003.07449/) (2020)
- [Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis](https://mlsanity.com/paper/2003.11571/) (2020)

## Built on by (most cited first, 30 of 31)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis](https://mlsanity.com/paper/2208.13753/) (2022)
- [LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation](https://mlsanity.com/paper/2302.08908/) (2023)
- [Controllable Text-to-Image Generation with GPT-4](https://mlsanity.com/paper/2305.18583/) (2023)
- [GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation](https://mlsanity.com/paper/2306.04607/) (2023)
- [Photoswap: Personalized Subject Swapping in Images](https://mlsanity.com/paper/2305.18286/) (2023)
- [Modeling Image Composition for Complex Scene Generation](https://mlsanity.com/paper/2206.00923/) (2022)
- [DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception](https://mlsanity.com/paper/2403.13304/) (2024)
- [LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts](https://mlsanity.com/paper/2308.06713/) (2023)
- [Audio Generation with Multiple Conditional Diffusion Model](https://mlsanity.com/paper/2308.11940/) (2023)
- [ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models](https://mlsanity.com/paper/2405.15199/) (2024)
- [EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model](https://mlsanity.com/paper/2306.11496/) (2023)
- [SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing](https://mlsanity.com/paper/2404.05717/) (2024)
- [Editable Image Elements for Controllable Synthesis](https://mlsanity.com/paper/2404.16029/) (2024)
- [Transformer-based Image Generation from Scene Graphs](https://mlsanity.com/paper/2303.04634/) (2023)
- [Vision + Language Applications: A Survey](https://mlsanity.com/paper/2305.14598/) (2023)
- [Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation](https://mlsanity.com/paper/2409.04847/) (2024)
- [Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouching](https://mlsanity.com/paper/2408.13858/) (2024)
- [GVDIFF: Grounded Text-to-Video Generation with Diffusion Models](https://mlsanity.com/paper/2407.01921/) (2024)
- [Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesis](https://mlsanity.com/paper/2510.18229/) (2025)
- [Video Generation Beyond a Single Clip](https://mlsanity.com/paper/2304.07483/) (2023)
- [MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis](https://mlsanity.com/paper/2305.05992/) (2023)
- [STAY Diffusion: Styled Layout Diffusion Model for Diverse Layout-to-Image Generation](https://mlsanity.com/paper/2503.12213/) (2025)
- [RMFlow: Refined Mean Flow by a Noise-Injection Step for Multimodal Generation](https://mlsanity.com/paper/2602.00849/) (2026)
- [OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System](https://mlsanity.com/paper/2303.00501/) (2023)
- [Image Synthesis with Graph Conditioning: CLIP-Guided Diffusion Models for Scene Graphs](https://mlsanity.com/paper/2401.14111/) (2024)
- [Tuning-Free Visual Customization via View Iterative Self-Attention Control](https://mlsanity.com/paper/2406.06258/) (2024)
- [Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation](https://mlsanity.com/paper/2501.09194/) (2025)
- [Contextualized Diffusion Models for Text-Guided Image and Video Generation](https://mlsanity.com/paper/2402.16627/) (2024)

## Abstract (arXiv)

The use of coarse-grained layouts for controllable synthesis of complex scene images via deep generative models has recently gained popularity. However, results of current approaches still fall short of their promise of high-resolution synthesis. We hypothesize that this is mostly due to the highly engineered nature of these approaches which often rely on auxiliary losses and intermediate steps such as mask generators. In this note, we present an orthogonal approach to this task, where the generative model is based on pure likelihood training without additional objectives. To do so, we first optimize a powerful compression model with adversarial training which learns to reconstruct its inputs via a discrete latent bottleneck and thereby effectively strips the latent representation of high-frequency details such as texture. Subsequently, we train an autoregressive transformer model to learn the distribution of the discrete image representations conditioned on a tokenized version of the layouts. Our experiments show that the resulting system is able to synthesize high-quality images consistent with the given layouts. In particular, we improve the state-of-the-art FID score on COCO-Stuff and on Visual Genome by up to 19% and 53% and demonstrate the synthesis of images up to 512 x 512 px on COCO and Open Images.