mlsanityData to 14 Oct 2026

CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions

arXiv 2112.05219 ↗First posted 9 Dec 2021cs.CV cs.GRcited by 78

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The success of StyleGAN has enabled unprecedented semantic editing capabilities, on both synthesized and real images. However, such editing operations are either trained with semantic supervision or described using human guidance. In another development, the CLIP architecture has been trained with internet-scale image and text pairings and has been shown to be useful in several zero-shot learning settings. In this work, we investigate how to effectively link the pretrained latent spaces of StyleGAN and CLIP, which in turn allows us to automatically extract semantically labeled edit directions from StyleGAN, finding and naming meaningful edit operations without any additional human guidance. Technically, we propose two novel building blocks; one for finding interesting CLIP directions and one for labeling arbitrary directions in CLIP latent space. The setup does not assume any pre-determined labels and hence we do not require any additional supervised text/attributes to build the editing framework. We evaluate the effectiveness of the proposed method and demonstrate that extraction of disentangled labeled StyleGAN edit directions is indeed possible, and reveals interesting and non-trivial edit directions.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 47 of its 60 references that are in the map. First posted on arXiv 9 Dec 2021.

Builds on

47 of 60 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  7. 2017Improved Training of Wasserstein GANscited by 2,235
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
35 more
  1. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  2. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  3. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,098
  4. 2016Least Squares Generative Adversarial Networkscited by 1,001
  5. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  8. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  9. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  10. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  11. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  12. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  13. 2020Closed-Form Factorization of Latent Semantics in GANscited by 408
  14. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 394
  15. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 349
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  17. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  18. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 289
  19. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 268
  20. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  21. 2019On the "steerability" of generative adversarial networkscited by 248
  22. 2021ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinementcited by 248
  23. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  24. 2021GNeRF: GAN-based Neural Radiance Field without Posed Cameracited by 153
  25. 2019Seeing What a GAN Cannot Generatecited by 148
  26. 2020Learning Visual Representations with Caption Annotationscited by 123
  27. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 92
  28. 2020Improved StyleGAN Embedding: Where are the Good Latents?cited by 88
  29. 2021Repurposing GANs for One-shot Semantic Part Segmentationcited by 88
  30. 2021Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networkscited by 68
  31. 2019Emergence of Object Segmentation in Perturbed Generative Modelscited by 57
  32. 2021Labels4Free: Unsupervised Segmentation using StyleGANcited by 53
  33. 2021Sketch Your Own GANcited by 37
  34. 2015Measuring Word Significance using Distributed Representations of Wordscited by 21
  35. 2020MichiGAN: Multi-Input-Conditioned Hair Image Generation for Portrait Editingcited by 18

Built on by

Cited by year

Citing papers per year: 2022: 21; 2023: 37; 2024: 16; 2025: 4.

78 papers.

Sort:

By area:

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  4. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  5. 2023Zero-shot Image-to-Image Translationcited by 505
  6. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  7. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  8. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  9. 2023Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Modelscited by 99
  10. 2023Improving Tuning-Free Real Image Editing with Proximal Guidancecited by 87
  11. 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
  12. 2023Face0: Instantaneously Conditioning a Text-to-Image Model on a Facecited by 78
38 more
  1. 2023Orthogonal Adaptation for Modular Customization of Diffusion Modelscited by 70
  2. 2022ClipFace: Text-guided Editing of Textured 3D Morphable Modelscited by 60
  3. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesiscited by 59
  4. 2022Third Time's the Charm? Image and Video Editing with StyleGAN3cited by 54
  5. 2023CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Modelcited by 47
  6. 2024Image Sculpting: Precise Object Editing with 3D Geometry Controlcited by 45
  7. 2022UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Imagecited by 42
  8. 2023DreamTuner: Single Image is Enough for Subject-Driven Generationcited by 41
  9. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 41
  10. 2023LayerDiffusion: Layered Controlled Image Editing with Diffusion Modelscited by 40
  11. 20233DAvatarGAN: Bridging Domains for Personalized Editable Avatarscited by 38
  12. 2022State-of-the-Art in the Architecture, Methods and Applications of StyleGANcited by 34
  13. 2023TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervisioncited by 28
  14. 2023ViCo: Plug-and-play Visual Condition for Personalized Text-to-image Generationcited by 23
  15. 2023iEdit: Localised Text-guided Image Editing with Weak Supervisioncited by 20
  16. 2023Local 3D Editing via 3D Distillation of CLIP Knowledgecited by 20
  17. 2023PFB-Diff: Progressive Feature Blending Diffusion for Text-driven Image Editingcited by 20
  18. 2024LoRACLR: Contrastive Adaptation for Customization of Diffusion Modelscited by 20
  19. 2023Hierarchical Diffusion Autoencoders and Disentangled Image Manipulationcited by 18
  20. 2023TextIR: A Simple Framework for Text-based Editable Image Restorationcited by 17
  21. 2022clip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIPcited by 16
  22. 2024Make Me Happier: Evoking Emotions Through Image Diffusion Modelscited by 16
  23. 2024ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extractioncited by 16
  24. 2023Free-Editor: Zero-shot Text-driven 3D Scene Editingcited by 14
  25. 2025IP-Composer: Semantic Composition of Visual Conceptscited by 14
  26. 2022Bridging CLIP and StyleGAN through Latent Alignment for Image Editingcited by 12
  27. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  28. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  29. 2024FastEdit: Fast Text-Guided Single-Image Editing via Semantic-Aware Diffusion Fine-Tuningcited by 10
  30. 2023CLIPXPlore: Coupled CLIP and Shape Spaces for 3D Shape Explorationcited by 9
  31. 2023Adaptive Nonlinear Latent Transformation for Conditional Face Editingcited by 9
  32. 2023Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generationcited by 9
  33. 2022IA-FaceS: A Bidirectional Method for Semantic Face Editingcited by 8
  34. 2022Video2StyleGAN: Disentangling Local and Global Variations in a Videocited by 8
  35. 2023Semantify: Simplifying the Control of 3D Morphable Models using CLIPcited by 8
  36. 2024LLMRA: Multi-modal Large Language Model based Restoration Assistantcited by 8
  37. 2022End-to-End Visual Editing with a Generatively Pre-Trained Artistcited by 7
  38. 2023Align, Adapt and Inject: Sound-guided Unified Image Generationcited by 6

50 most cited of 78 All 78 (JSON)