# Text to Image Generation with Semantic-Spatial Aware GAN

arXiv: https://arxiv.org/abs/2104.00567 · HTML: https://mlsanity.com/paper/2104.00567/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning](https://mlsanity.com/paper/1611.05594/) (2016)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [Controllable Text-to-Image Generation](https://mlsanity.com/paper/1909.07083/) (2019)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [Stabilizing Training of Generative Adversarial Networks through Regularization](https://mlsanity.com/paper/1705.09367/) (2017)
- [Object-driven Text-to-Image Synthesis via Adversarial Training](https://mlsanity.com/paper/1902.10740/) (2019)
- [Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis](https://mlsanity.com/paper/1910.06809/) (2019)
- [Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis](https://mlsanity.com/paper/1801.05091/) (2018)
- [Specifying Object Attributes and Relations in Interactive Scene Generation](https://mlsanity.com/paper/1909.05379/) (2019)
- [Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network](https://mlsanity.com/paper/1802.09178/) (2018)
- [Semantics Disentangling for Text-to-Image Generation](https://mlsanity.com/paper/1904.01480/) (2019)

## Built on by (most cited first, 30 of 53)

- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2304.02642/) (2023)
- [Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion](https://mlsanity.com/paper/2403.16387/) (2024)
- [Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis](https://mlsanity.com/paper/2208.13753/) (2022)
- [Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2304.01816/) (2023)
- [Shape-aware Text-driven Layered Video Editing](https://mlsanity.com/paper/2301.13173/) (2023)
- [RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model](https://mlsanity.com/paper/2309.00810/) (2023)
- [SceneComposer: Any-Level Semantic Image Synthesis](https://mlsanity.com/paper/2211.11742/) (2022)
- [EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2401.04608/) (2024)
- [Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generation](https://mlsanity.com/paper/2305.13607/) (2023)
- [Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similarities](https://mlsanity.com/paper/2407.20337/) (2024)
- [A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation](https://mlsanity.com/paper/2304.02858/) (2023)
- [Attribute-Centric Compositional Text-to-Image Generation](https://mlsanity.com/paper/2301.01413/) (2023)
- [Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysis](https://mlsanity.com/paper/2304.10278/) (2023)
- [UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries](https://mlsanity.com/paper/2507.23372/) (2025)
- [TP2O: Creative Text Pair-to-Object Generation using Balance Swap-Sampling](https://mlsanity.com/paper/2310.01819/) (2023)
- [Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects](https://mlsanity.com/paper/2311.02332/) (2023)
- [Recurrent Affine Transformation for Text-to-image Synthesis](https://mlsanity.com/paper/2204.10482/) (2022)
- [HumanDiffusion: a Coarse-to-Fine Alignment Diffusion Framework for Controllable Text-Driven Person Image Generation](https://mlsanity.com/paper/2211.06235/) (2022)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [Vision + Language Applications: A Survey](https://mlsanity.com/paper/2305.14598/) (2023)
- [Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community](https://mlsanity.com/paper/2402.09872/) (2024)
- [An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editing](https://mlsanity.com/paper/2508.17435/) (2025)
- [DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing](https://mlsanity.com/paper/2312.03772/) (2023)
- [DIFFNAT: Improving Diffusion Image Quality Using Natural Image Statistics](https://mlsanity.com/paper/2311.09753/) (2023)
- [Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey](https://mlsanity.com/paper/2412.17616/) (2024)
- [T-Person-GAN: Text-to-Person Image Generation with Identity-Consistency and Manifold Mix-Up](https://mlsanity.com/paper/2208.12752/) (2022)
- [EMMA: Concept Erasure Benchmark with Comprehensive Semantic Metrics and Diverse Categories](https://mlsanity.com/paper/2512.17320/) (2025)
- [Paint it Black: Generating paintings from text descriptions](https://mlsanity.com/paper/2302.08808/) (2023)
- [T2UE: Generating Unlearnable Examples from Text Descriptions](https://mlsanity.com/paper/2508.03091/) (2025)

## Abstract (arXiv)

Text-to-image synthesis (T2I) aims to generate photo-realistic images which are semantically consistent with the text descriptions. Existing methods are usually built upon conditional generative adversarial networks (GANs) and initialize an image from noise with sentence embedding, and then refine the features with fine-grained word embedding iteratively. A close inspection of their generated images reveals a major limitation: even though the generated image holistically matches the description, individual image regions or parts of somethings are often not recognizable or consistent with words in the sentence, e.g. "a white crown". To address this problem, we propose a novel framework Semantic-Spatial Aware GAN for synthesizing images from input text. Concretely, we introduce a simple and effective Semantic-Spatial Aware block, which (1) learns semantic-adaptive transformation conditioned on text to effectively fuse text features and image features, and (2) learns a semantic mask in a weakly-supervised way that depends on the current text-image fusion process in order to guide the transformation spatially. Experiments on the challenging COCO and CUB bird datasets demonstrate the advantage of our method over the recent state-of-the-art approaches, regarding both visual fidelity and alignment with input text description.