mlsanityData to 7 Oct 2026

GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis

arXiv 2301.12959 ↗First posted 30 Jan 2023cs.CV cs.AIcited by 84

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Synthesizing high-fidelity complex images from text is challenging. Based on large pretraining, the autoregressive and diffusion models can synthesize photo-realistic images. Although these large models have shown notable progress, there remain three flaws. 1) These models require tremendous training data and parameters to achieve good performance. 2) The multi-step generation design slows the image synthesis process heavily. 3) The synthesized visual features are difficult to control and require delicately designed prompts. To enable high-quality, efficient, fast, and controllable text-to-image synthesis, we propose Generative Adversarial CLIPs, namely GALIP. GALIP leverages the powerful pretrained CLIP model both in the discriminator and generator. Specifically, we propose a CLIP-based discriminator. The complex scene understanding ability of CLIP enables the discriminator to accurately assess the image quality. Furthermore, we propose a CLIP-empowered generator that induces the visual concepts from CLIP through bridge features and prompts. The CLIP-integrated generator and discriminator boost training efficiency, and as a result, our model only requires about 3% training data and 6% learnable parameters, achieving comparable results to large pretrained autoregressive and diffusion models. Moreover, our model achieves 120 times faster synthesis speed and inherits the smooth latent space from GAN. The extensive experimental results demonstrate the excellent performance of our GALIP. Code is available at https://github.com/tobran/GALIP.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 51 of its 61 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

51 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
35 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2022Visual Prompt Tuningcited by 1,674
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  6. 2018Self-Attention Generative Adversarial Networkscited by 809
  7. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  8. 2016Generative Adversarial Text to Image Synthesiscited by 735
  9. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  10. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  11. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  12. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  13. 2019Deep Learning for Image Super-resolution: A Surveycited by 368
  14. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  15. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  16. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  17. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  18. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  19. 2021Projected GANs Converge Fastercited by 204
  20. 2019Controllable Text-to-Image Generationcited by 187
  21. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  22. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  23. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  24. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  25. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  26. 2021M6: A Chinese Multimodal Pretrainercited by 91
  27. 2019Semantics Disentangling for Text-to-Image Generationcited by 61
  28. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  29. 2021DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesiscited by 45
  30. 2021Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Modelcited by 34
  31. 2023Attribute-Centric Compositional Text-to-Image Generationcited by 19
  32. 2022DE-Net: Dynamic Text-guided Image Editing Adversarial Networkscited by 12
  33. 2022Recurrent Affine Transformation for Text-to-image Synthesiscited by 11
  34. 2020TIME: Text and Image Mutual-Translation Adversarial Networkscited by 9
  35. 2022Layout-Bridging Text-to-Image Synthesiscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 35; 2025: 16; 2026: 10.

84 papers.

Sort:

By area:

  1. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  2. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  3. 2024Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Modelscited by 164
  4. 2023Raising the Bar of AI-generated Image Detection with CLIPcited by 163
  5. 2023Intriguing properties of synthetic images: from generative adversarial networks to diffusion modelscited by 121
  6. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  7. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  8. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  9. 2024LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insightscited by 65
  10. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Modelscited by 62
  11. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  12. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
38 more
  1. 2023Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Modelscited by 50
  2. 2024Zero-Shot Detection of AI-Generated Imagescited by 50
  3. 2024Community Forensics: Using Thousands of Generators to Train Fake Image Detectorscited by 50
  4. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  5. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  6. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  7. 2025Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Modelcited by 46
  8. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  9. 2024WildFake: A Large-scale Challenging Dataset for AI-Generated Images Detectioncited by 31
  10. 2024Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similaritiescited by 27
  11. 2024LEDiff: Latent Exposure Diffusion for HDR Generationcited by 22
  12. 2023Diffusion Noise Feature: Accurate and Fast Generated Image Detectioncited by 21
  13. 2023Graph Transformer GANs for Graph-Constrained House Generationcited by 19
  14. 2024StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completioncited by 19
  15. 2024CLIP-Guided Generative Networks for Transferable Targeted Adversarial Attackscited by 19
  16. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIPcited by 19
  17. 2024Mimir: Improving Video Diffusion Models for Precise Text Understandingcited by 16
  18. 2023The Age of Synthetic Realities: Challenges and Opportunitiescited by 15
  19. 2024Which Model Generated This Image? A Model-Agnostic Approach for Origin Attributioncited by 15
  20. 2023Kiki or Bouba? Sound Symbolism in Vision-and-Language Modelscited by 14
  21. 2024A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIPcited by 14
  22. 2023BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusioncited by 13
  23. 2023Exploring Sparse MoE in GANs for Text-conditioned Image Synthesiscited by 13
  24. 2024Synthetic Image Verification in the Era of Generative AI: What Works and What Isn't There Yetcited by 13
  25. 2025SMRABooth: Subject and Motion Representation Alignment for Customized Video Generationcited by 9
  26. 2023Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesiscited by 8
  27. 2025Scaling Up AI-Generated Image Detection with Generator-Aware Prototypescited by 8
  28. 2024ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancementscited by 7
  29. 2025Distributional Vision-Language Alignment by Cauchy-Schwarz Divergencecited by 7
  30. 2023TPA3D: Triplane Attention for Fast Text-to-3D Generationcited by 6
  31. 2023Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluationcited by 5
  32. 2024Graph Transformer GANs with Graph Masked Modeling for Architectural Layout Generationcited by 5
  33. 2024Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applicationscited by 5
  34. 2024SIA-OVD: Shape-Invariant Adapter for Bridging the Image-Region Gap in Open-Vocabulary Detectioncited by 5
  35. 2023Efficient Text-Guided 3D-Aware Portrait Generation with Score Distillation Sampling on Distributioncited by 4
  36. 2023T2IW: Joint Text to Image & Watermark Generationcited by 4
  37. 2025BOOD: Boundary-based Out-Of-Distribution Data Generationcited by 4
  38. 2024Trustworthy Image Semantic Communication with GenAI: Explainablity, Controllability, and Efficiencycited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025un2CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
  2. 2024Text-to-Image Generation Via Energy-Based CLIP
  3. 2022clip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIP
  4. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts
  5. 2023CLIPAG: Towards Generator-Free Text-to-Image Generation
  6. 2023Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion
  7. 2024Diffusion Feedback Helps CLIP See Better
  8. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis