mlsanityData to 7 Oct 2026

CLIP-VQDiffusion : Langauge Free Training of Text To Image generation using CLIP and vector quantized diffusion model

arXiv 2403.14944 ↗First posted 22 Mar 2024cs.CVcited by 2

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

There has been a significant progress in text conditional image generation models. Recent advancements in this field depend not only on improvements in model structures, but also vast quantities of text-image paired datasets. However, creating these kinds of datasets is very costly and requires a substantial amount of labor. Famous face datasets don't have corresponding text captions, making it difficult to develop text conditional image generation models on these datasets. Some research has focused on developing text to image generation models using only images without text captions. Here, we propose CLIP-VQDiffusion, which leverage the pretrained CLIP model to provide multimodal text-image representations and strong image generation capabilities. On the FFHQ dataset, our model outperformed previous state-of-the-art methods by 4.4% in clipscore and generated very realistic images even when the text was both in and out of distribution. The pretrained models and codes will soon be available at https://github.com/INFINIQ-AI1/CLIPVQDiffusion

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 28 of its 31 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

28 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022clip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIP5 Oct
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
14 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2021Scaling Vision Transformerscited by 993
  3. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  4. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  5. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  7. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  8. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  9. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  10. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  11. 2022Improved Vector Quantized Diffusion Modelscited by 62
  12. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  13. 2023Regularized Vector Quantization for Tokenized Image Synthesiscited by 46
  14. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024G2D2: Gradient-Guided Discrete Diffusion for Inverse Problem Solvingcited by 4
  2. 2025DiffListener: Discrete Diffusion Model for Listener Generationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents
  2. 2024Diffusion Feedback Helps CLIP See Better
  3. 2023Controlled and Conditional Text to Image Generation with Diffusion Prior
  4. 2022CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning
  5. 2023The CLIP Model is Secretly an Image-to-Prompt Converter
  6. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
  7. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Model
  8. 2024Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation