mlsanityData to 13 Oct 2026

VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance

arXiv 2204.08583 ↗First posted 18 Apr 2022cs.CVcited by 285

Not on the map. Why?

Cited by 89 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating and editing images from open domain text prompts is a challenging task that heretofore has required expensive and specially trained models. We demonstrate a novel methodology for both tasks which is capable of producing images of high visual quality from text prompts of significant semantic complexity without any training by using a multimodal encoder to guide image generations. We demonstrate on a variety of tasks how using CLIP [37] to guide VQGAN [11] produces higher visual quality outputs than prior, less flexible approaches like DALL-E [38], GLIDE [33] and Open-Edit [24], despite not being trained for the tasks presented. Our code is available in a public repository.

Key facts. Cited by 285 papers in the map (data through 13 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

42 of 57 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,713
  4. 2017Neural Discrete Representation Learningcited by 4,680
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,605
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,360
  11. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  12. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 926
30 more
  1. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 659
  4. 2021Merging Models with Fisher-Weighted Averagingcited by 617
  5. 2021Editing Factual Knowledge in Language Modelscited by 551
  6. 2021Fast Model Editing at Scalecited by 493
  7. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
  8. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 232
  9. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 211
  10. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  11. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 187
  12. 2021The Values Encoded in Machine Learning Researchcited by 172
  13. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  14. 2020Rewriting a Deep Generative Modelcited by 106
  15. 2020The Cost of Training NLP Models: A Concise Overviewcited by 98
  16. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  17. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  18. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  19. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  20. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  21. 2020SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objectscited by 43
  22. 2021Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Conceptscited by 40
  23. 2020Semantic Pyramid for Image Generationcited by 34
  24. 2022FlexIT: Towards Flexible Semantic Image Translationcited by 28
  25. 2020Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructionscited by 25
  26. 2021AffectGAN: Affect-Based Generative Art Driven by Semanticscited by 10
  27. 2021Telling Creative Stories Using Generative Visual Aidscited by 3
  28. 2022Music2Video: Automatic Generation of Music Video with fusion of audio and textcited by 3
  29. 2020MUSE: Textual Attributes Guided Portrait Painting Generationcited by 2
  30. 2021Insiders and Outsiders in Research on Machine Learning and Societycited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 47; 2023: 108; 2024: 84; 2025: 28; 2026: 18.

285 papers.

Sort:

By area:

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,294
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,604
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,421
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,240
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,174
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,112
  9. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 903
  10. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 725
  11. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 701
  12. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 645
38 more
  1. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 619
  2. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 583
  3. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 563
  4. 2022Blended Latent Diffusioncited by 432
  5. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 391
  6. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 344
  7. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  8. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 301
  9. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 299
  10. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 291
  11. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 267
  12. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  13. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 237
  14. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 235
  15. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 234
  16. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  17. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  18. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  19. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  20. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 183
  21. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 155
  22. 2022Diffusion-based Image Translation using Disentangled Style and Content Representationcited by 149
  23. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  24. 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervisioncited by 145
  25. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 130
  26. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  27. 2023Delta Denoising Scorecited by 121
  28. 2023Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Modelscited by 98
  29. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  30. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 96
  31. 2023Stay on topic with Classifier-Free Guidancecited by 90
  32. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  33. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  34. 2023ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Modelscited by 75
  35. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generationcited by 74
  36. 2023StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editingcited by 68
  37. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 68
  38. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Modelscited by 63

50 most cited of 285 All 285 (JSON)