# SceneComposer: Any-Level Semantic Image Synthesis

arXiv: https://arxiv.org/abs/2211.11742 · HTML: https://mlsanity.com/paper/2211.11742/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 75 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 30 of 67)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)

## Built on by (most cited first, 30 of 43)

- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance](https://mlsanity.com/paper/2306.00943/) (2023)
- [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025)
- [A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesis](https://mlsanity.com/paper/2306.14544/) (2023)
- [Zero-shot spatial layout conditioning for text-to-image diffusion models](https://mlsanity.com/paper/2306.13754/) (2023)
- [Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D](https://mlsanity.com/paper/2312.02190/) (2023)
- [Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2304.03869/) (2023)
- [Photoswap: Personalized Subject Swapping in Images](https://mlsanity.com/paper/2305.18286/) (2023)
- [TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation](https://mlsanity.com/paper/2406.08656/) (2024)
- [Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback](https://mlsanity.com/paper/2307.04749/) (2023)
- [A Survey of Diffusion Based Image Generation Models: Issues and Their Solutions](https://mlsanity.com/paper/2308.13142/) (2023)
- [LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.12342/) (2023)
- [SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesis](https://mlsanity.com/paper/2311.03355/) (2023)
- [Fine-grained Controllable Video Generation via Object Appearance and Context](https://mlsanity.com/paper/2312.02919/) (2023)
- [Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation](https://mlsanity.com/paper/2408.14819/) (2024)
- [PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor](https://mlsanity.com/paper/2303.17546/) (2023)
- [SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing](https://mlsanity.com/paper/2404.05717/) (2024)
- [PreciseCam: Precise Camera Control for Text-to-Image Generation](https://mlsanity.com/paper/2501.12910/) (2025)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control](https://mlsanity.com/paper/2506.00596/) (2025)
- [Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation](https://mlsanity.com/paper/2308.06027/) (2023)
- [Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Models](https://mlsanity.com/paper/2402.17910/) (2024)
- [MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2404.09977/) (2024)
- [Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis](https://mlsanity.com/paper/2307.12084/) (2023)
- [Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation](https://mlsanity.com/paper/2403.16394/) (2024)
- [Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis](https://mlsanity.com/paper/2401.09048/) (2024)
- [PARASOL: Parametric Style Control for Diffusion Image Synthesis](https://mlsanity.com/paper/2303.06464/) (2023)
- [Text-Anchored Score Composition: Tackling Condition Misalignment in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.14408/) (2023)
- [Layout-to-Image Generation with Localized Descriptions using ControlNet with Cross-Attention Control](https://mlsanity.com/paper/2402.13404/) (2024)
- [Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition](https://mlsanity.com/paper/2503.10037/) (2025)

## Abstract (arXiv)

We propose a new framework for conditional image synthesis from semantic layouts of any precision levels, ranging from pure text to a 2D semantic canvas with precise shapes. More specifically, the input layout consists of one or more semantic regions with free-form text descriptions and adjustable precision levels, which can be set based on the desired controllability. The framework naturally reduces to text-to-image (T2I) at the lowest level with no shape information, and it becomes segmentation-to-image (S2I) at the highest level. By supporting the levels in-between, our framework is flexible in assisting users of different drawing expertise and at different stages of their creative workflow. We introduce several novel techniques to address the challenges coming with this new setup, including a pipeline for collecting training data; a precision-encoded mask pyramid and a text feature map representation to jointly encode precision level, semantics, and composition information; and a multi-scale guided diffusion model to synthesize images. To evaluate the proposed method, we collect a test dataset containing user-drawn layouts with diverse scenes and styles. Experimental results show that the proposed method can generate high-quality images following the layout at given precision, and compares favorably against existing methods. Project page urlhttps://zengxianyu.github.io/scenec/