mlsanityData to 7 Oct 2026

LAFITE: Towards Language-Free Training for Text-to-Image Generation

arXiv 2111.13792 ↗First posted 27 Nov 2021cs.CV cs.LGcited by 126

Not on the map. Why?

Cited by 59 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs. While image samples are often easily accessible, the associated text descriptions typically require careful human captioning, which is particularly time- and cost-consuming. In this paper, we propose the first work to train text-to-image generation models without any text data. Our method leverages the well-aligned multi-modal semantic space of the powerful pre-trained CLIP model: the requirement of text-conditioning is seamlessly alleviated via generating text features from image features. Extensive experiments are conducted to illustrate the effectiveness of the proposed method. We obtain state-of-the-art results in the standard text-to-image generation tasks. Importantly, the proposed language-free model outperforms most existing models trained with full image-text pairs. Furthermore, our method can be applied in fine-tuning pre-trained models, which saves both training time and cost in training text-to-image generation models. Our pre-trained model obtains competitive results in zero-shot text-to-image generation on the MS-COCO dataset, yet with around only 1% of the model size and training data size relative to the recently proposed large DALL-E model.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 27 Nov 2021.

Builds on

46 of 59 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  12. 2013Auto-Encoding Variational Bayescited by 3,980
34 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  3. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2014Conditional Generative Adversarial Netscited by 2,152
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  13. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  14. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  15. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  16. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  17. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  18. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  21. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 331
  22. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  23. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  24. 2019Connecting Vision and Language with Localized Narrativescited by 225
  25. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  26. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  27. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  28. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  29. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  30. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
  31. 2021Training GANs with Stronger Augmentations via Contrastive Discriminatorcited by 45
  32. 2017ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matchingcited by 32
  33. 2021Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIPcited by 21
  34. 2020Style Intervention: How to Achieve Spatial Disentanglement with Style-based Generators?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 21; 2023: 43; 2024: 42; 2025: 13; 2026: 5.

126 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  5. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  6. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  8. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  9. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  10. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  11. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  12. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
38 more
  1. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  2. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  3. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  4. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  5. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  6. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  7. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  8. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  9. 2023SneakyPrompt: Jailbreaking Text-to-image Generative Modelscited by 175
  10. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
  11. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  12. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  13. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  14. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  15. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
  16. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  17. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 84
  18. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 66
  19. 2023AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondencecited by 61
  20. 2024Rectified Diffusion: Straightness Is Not Your Need in Rectified Flowcited by 59
  21. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesiscited by 55
  22. 2024CRS-Diff: Controllable Remote Sensing Image Generation with Diffusion Modelcited by 54
  23. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  24. 2025Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Modelcited by 46
  25. 2023MobileDiffusion: Instant Text-to-Image Generation on Mobile Devicescited by 42
  26. 2024Scaling Diffusion Models to Real-World 3D LiDAR Scene Completioncited by 40
  27. 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generationcited by 39
  28. 2022Shifted Diffusion for Text-to-image Generationcited by 37
  29. 2022Self-Guided Diffusion Modelscited by 36
  30. 2022High-Fidelity Guided Image Synthesis with Latent Diffusion Modelscited by 36
  31. 2024Improving Diffusion-Based Image Synthesis with Context Predictioncited by 32
  32. 2024Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Predictioncited by 31
  33. 2023BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstructioncited by 30
  34. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferencescited by 30
  35. 2023LD-ZNet: A Latent Diffusion Approach for Text-Based Image Segmentationcited by 29
  36. 2022Txt2Img-MHN: Remote Sensing Image Generation from Text Using Modern Hopfield Networkscited by 28
  37. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  38. 2023Textual Prompt Guided Image Restorationcited by 27

50 most cited of 126 All 126 (JSON)