mlsanityData to 7 Oct 2026

Text-to-Image Cross-Modal Generation: A Systematic Review

arXiv 2401.11631 ↗First posted 21 Jan 2024cs.CV cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We review research on generating visual data from text from the angle of "cross-modal generation." This point of view allows us to draw parallels between various methods geared towards working on input text and producing visual output, without limiting the analysis to narrow sub-areas. It also results in the identification of common templates in the field, which are then compared and contrasted both within pools of similar methods and across lines of research. We provide a breakdown of text-to-image generation into various flavors of image-from-text methods, video-from-text methods, image editing, self-supervised and graph-based approaches. In this discussion, we focus on research papers published at 8 leading machine learning conferences in the years 2016-2022, also incorporating a number of relevant papers not matching the outlined search criteria. The conducted review suggests a significant increase in the number of papers published in the area and highlights research gaps and potential lines of investigation. To our knowledge, this is the first review to systematically look at text-to-image generation from the perspective of "cross-modal generation."

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 115 of its 167 references that are in the map. First posted on arXiv 21 Jan 2024.

Builds on

115 of 167 references.

References · 115

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  10. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  11. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2016Improved Techniques for Training GANscited by 2,938
  13. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  14. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  15. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  16. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  17. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  18. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  19. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  20. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  21. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  22. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  23. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  24. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  25. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  26. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  27. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  28. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  29. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  30. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  31. 2016Pixel Recurrent Neural Networkscited by 814
  32. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 751
  33. 2016Generative Adversarial Text to Image Synthesiscited by 735
  34. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  35. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  36. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  37. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  38. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624

50 most cited of 115 All 115 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  2. 2024Graph Transformers: A Surveycited by 32
  3. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  4. 2025Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessmentcited by 2
  5. 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Modelscited by 1