Data to 7 Oct 2026

Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

arXiv 1801.05091 ↗First posted 16 Jan 2018cs.CVcited by 101

Not on the map. Why?

Abstract (arXiv)

We propose a novel hierarchical approach for text-to-image synthesis by inferring semantic layout. Instead of learning a direct mapping from text to image, our algorithm decomposes the generation process into multiple steps, in which it first constructs a semantic layout from the text by the layout generator and converts the layout to an image by the image generator. The proposed layout generator progressively constructs a semantic layout in a coarse-to-fine manner by generating object bounding boxes and refining each box by estimating object shapes inside the box. The image generator synthesizes an image conditioned on the inferred semantic layout, which provides a useful semantic structure of an image matching with the text description. Our model not only generates semantically more meaningful images, but also allows automatic annotation of generated images and user-controlled generation process by modifying the generated scene layout. We demonstrate the capability of the proposed model on challenging MS-COCO dataset and show that the model can substantially improve the image quality, interpretability of output and semantic alignment to input text over existing approaches.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 26; 2023: 30; 2024: 6; 2025: 3; 2026: 4.

101 papers.

Sort:

By area:

2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 258
2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230

38 more

2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 84
2021SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Datacited by 78
2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75
2021Learning to Compose Visual Relationscited by 72
2023TaleCrafter: Interactive Story Visualization with Multiple Characterscited by 71
2023Freestyle Layout-to-Image Synthesiscited by 67
2022Synthetic Data -- what, why and how?cited by 66
2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 66
2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
2021SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editingcited by 58
2021Text to Image Generation with Semantic-Spatial Aware GANcited by 53
2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52
2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
2021Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualizationcited by 51
2021DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesiscited by 45
2022Modeling Image Composition for Complex Scene Generationcited by 43
2022SceneComposer: Any-Level Semantic Image Synthesiscited by 43
2021Context-Aware Layout to Image Generation with Enhanced Object Appearancecited by 41
2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
2022SGUIE-Net: Semantic Attention Guided Underwater Image Enhancement with Multi-Scale Perceptioncited by 32
2023High-Fidelity 3D Face Generation from Natural Language Descriptionscited by 32
2022VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programmingcited by 31
2023DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planningcited by 28
2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
2021Compositional Transformers for Scene Generationcited by 24
2022Interactive Image Synthesis with Panoptic Layout Generationcited by 17
2022ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Designcited by 16
2023DiffCloth: Diffusion Based Garment Synthesis and Manipulation via Structural Cross-modal Semantic Alignmentcited by 15
2023A Parse-Then-Place Approach for Generating Graphic Layouts from Textual Descriptionscited by 15
2021ClimateGAN: Raising Climate Change Awareness by Generating Images of Floodscited by 14

50 most cited of 101 All 101 (JSON)