mlsanityData to 7 Oct 2026

Multi-Tailed, Multi-Headed, Spatial Dynamic Memory refined Text-to-Image Synthesis

arXiv 2110.08143 ↗First posted 15 Oct 2021cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Synthesizing high-quality, realistic images from text-descriptions is a challenging task, and current methods synthesize images from text in a multi-stage manner, typically by first generating a rough initial image and then refining image details at subsequent stages. However, existing methods that follow this paradigm suffer from three important limitations. Firstly, they synthesize initial images without attempting to separate image attributes at a word-level. As a result, object attributes of initial images (that provide a basis for subsequent refinement) are inherently entangled and ambiguous in nature. Secondly, by using common text-representations for all regions, current methods prevent us from interpreting text in fundamentally different ways at different parts of an image. Different image regions are therefore only allowed to assimilate the same type of information from text at each refinement stage. Finally, current methods generate refinement features only once at each refinement stage and attempt to address all image aspects in a single shot. This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image. Our proposed method introduces three novel components to address these shortcomings: (1) An initial generation stage that explicitly generates separate sets of image features for each word n-gram. (2) A spatial dynamic memory module for refinement of images. (3) An iterative multi-headed mechanism to make it easier to improve upon multiple image aspects. Experimental results demonstrate that our Multi-Headed Spatial Dynamic Memory image refinement with our Multi-Tailed Word-level Initial Generation (MSMT-GAN) performs favourably against the previous state of the art on the CUB and COCO datasets.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

18 of 26 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  6. 2016Generative Adversarial Text to Image Synthesiscited by 735
  7. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  9. 2014Memory Networkscited by 354
  10. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  11. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  12. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
6 more
  1. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  2. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  3. 2019Controllable Text-to-Image Generationcited by 187
  4. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  5. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  6. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58

Built on by

Cited by year

Citing papers per year: 2022: 2.

2 papers.

  1. 2022DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generationcited by 13
  2. 2022SSD: Towards Better Text-Image Consistency Metric in Text-to-Image Generationcited by 2