mlsanityData to 7 Oct 2026

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

arXiv 2103.17249 ↗First posted 31 Mar 2021cs.CV cs.CL cs.GRcited by 990

Not on the map. Why?

Cited by 206 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Inspired by the ability of StyleGAN to generate highly realistic images in a variety of domains, much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images. However, discovering semantically meaningful latent manipulations typically involves painstaking human examination of the many degrees of freedom, or an annotated collection of images for each desired manipulation. In this work, we explore leveraging the power of recently introduced Contrastive Language-Image Pre-training (CLIP) models in order to develop a text-based interface for StyleGAN image manipulation that does not require such manual effort. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable text-based manipulation. Finally, we present a method for mapping a text prompts to input-agnostic directions in StyleGAN's style space, enabling interactive text-driven image manipulation. Extensive results and comparisons demonstrate the effectiveness of our approaches.

Key facts. Cited by 990 papers in the map (data through 7 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

47 of 63 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  12. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
35 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2014Conditional Generative Adversarial Netscited by 2,152
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  9. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2016Generative Adversarial Text to Image Synthesiscited by 735
  12. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  13. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  14. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  15. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  16. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  17. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  18. 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
  19. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 393
  20. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  21. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  22. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 331
  23. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
  24. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 267
  25. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  26. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  27. 2020Learning Visual Representations with Caption Annotationscited by 123
  28. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  29. 2021Paint by Wordcited by 107
  30. 2021Only a Matter of Style: Age Transformation Using a Style-Based Regression Modelcited by 92
  31. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  32. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  33. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
  34. 2020Text-to-Image Generation Grounded by Fine-Grained User Attentioncited by 36
  35. 2020Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approachcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 65; 2022: 242; 2023: 331; 2024: 185; 2025: 113; 2026: 54.

990 papers.

Sort:

By area:

  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  12. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
38 more
  1. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  2. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  3. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  4. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  5. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  6. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  7. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  8. 2021Extract Free Dense Labels from CLIPcited by 580
  9. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  10. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  11. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  12. 2021Image Segmentation Using Text and Image Promptscited by 509
  13. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  14. 2023Zero-shot Image-to-Image Translationcited by 503
  15. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  16. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  17. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  18. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  19. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  20. 2022Blended Latent Diffusioncited by 430
  21. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  22. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  23. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  24. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  25. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  26. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  27. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  28. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  29. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  30. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  31. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  32. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  33. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  34. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  35. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  36. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  37. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 290
  38. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285

50 most cited of 990 All 990 (JSON)