mlsanityData to 14 Oct 2026

CLIPAG: Towards Generator-Free Text-to-Image Generation

arXiv 2306.16805 ↗First posted 29 Jun 2023cs.CV cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Perceptually Aligned Gradients (PAG) refer to an intriguing property observed in robust image classification models, wherein their input gradients align with human perception and pose semantic meanings. While this phenomenon has gained significant research attention, it was solely studied in the context of unimodal vision-only architectures. In this work, we extend the study of PAG to Vision-Language architectures, which form the foundations for diverse image-text tasks and applications. Through an adversarial robustification finetuning of CLIP, we demonstrate that robust Vision-Language models exhibit PAG in contrast to their vanilla counterparts. This work reveals the merits of CLIP with PAG (CLIPAG) in several vision-language generative tasks. Notably, we show that seamlessly integrating CLIPAG in a "plug-n-play" manner leads to substantial improvements in vision-language generative applications. Furthermore, leveraging its PAG property, CLIPAG enables text-to-image generation without any generative model, which typically requires huge generators.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 49 of its 74 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

49 of 74 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2013Intriguing properties of neural networkscited by 5,155
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
37 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  4. 2016Improved Techniques for Training GANscited by 2,970
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  11. 2018Robustness May Be at Odds with Accuracycited by 880
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  13. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  14. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  15. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  16. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  17. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  18. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  19. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 363
  20. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  21. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  22. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  23. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  24. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 187
  25. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  26. 2022VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Modelscited by 112
  27. 2019On the Connection Between Adversarial Robustness and Saliency Map Interpretabilitycited by 80
  28. 2020SCATTER: Selective Context Attentional Scene Text Recognizercited by 69
  29. 2022CLIPasso: Semantically-Aware Object Sketchingcited by 57
  30. 2019Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?cited by 43
  31. 2023Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustnesscited by 20
  32. 2022Multimodal Semi-Supervised Learning for Text Recognitioncited by 16
  33. 2022Do Perceptually Aligned Gradients Imply Adversarial Robustness?cited by 15
  34. 2023Towards Models that Can See and Readcited by 15
  35. 2020On the Benefits of Models with Perceptually-Aligned Gradientscited by 13
  36. 2021BIGRoC: Boosting Image Generation via a Robust Classifiercited by 9
  37. 2023Class-Conditioned Transformation for Enhanced Robust Image Classificationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 3.

12 papers.

  1. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  2. 2025Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Conceptscited by 12
  3. 2025Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metricscited by 10
  4. 2025LeakyCLIP: Extracting Training Data from CLIPcited by 4
  5. 2024Text-to-Image Generation Via Energy-Based CLIPcited by 3
  6. 2024Enhancing Consistency-Based Image Generation via Adversarialy-Trained Classification and Energy-Based Discriminationcited by 2
  7. 2024L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancementcited by 2
  8. 2025Mind the Detail: Uncovering Clinically Relevant Image Details in Accelerated MRI with Semantically Diverse Reconstructionscited by 2
  9. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Modelscited by 1
  10. 2026Optimizing 4D Wires for Sparse 3D Abstractioncited by 0
  11. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0
  12. 2026Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  2. 2024CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance
  3. 2023Revisiting the Role of Language Priors in Vision-Language Models
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters
  5. 2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Models
  6. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs
  7. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Training
  8. 2024Diffusion Feedback Helps CLIP See Better