mlsanityData to 7 Oct 2026

Designing an Encoder for StyleGAN Image Manipulation

arXiv 2102.02766 ↗First posted 4 Feb 2021cs.CVcited by 542

Not on the map. Why?

Cited by 74 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, there has been a surge of diverse methods for performing image editing by employing pre-trained unconditional generators. Applying these methods on real images, however, remains a challenge, as it necessarily requires the inversion of the images into their latent space. To successfully invert a real image, one needs to find a latent code that reconstructs the input image accurately, and more importantly, allows for its meaningful manipulation. In this paper, we carefully study the latent space of StyleGAN, the state-of-the-art unconditional generator. We identify and analyze the existence of a distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space. We then suggest two principles for designing encoders in a manner that allows one to control the proximity of the inversions to regions that StyleGAN was originally trained on. We present an encoder based on our two principles that is specifically designed for facilitating editing on real images by balancing these tradeoffs. By evaluating its performance qualitatively and quantitatively on numerous challenging domains, including cars and horses, we show that our inversion method, followed by common editing techniques, achieves superior real-image editing quality, with only a small reconstruction accuracy drop.

Key facts. Cited by 542 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

37 of 55 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  9. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  10. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  11. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  12. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
25 more
  1. 2018Which Training Methods for GANs do actually Converge?cited by 672
  2. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  3. 2017The Perception-Distortion Tradeoffcited by 582
  4. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  5. 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
  6. 2021GAN Inversion: A Surveycited by 366
  7. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  8. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  9. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  10. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  11. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 331
  12. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
  13. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 267
  14. 2019On the "steerability" of generative adversarial networkscited by 247
  15. 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
  16. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  17. 2016Inverting The Generator Of A Generative Adversarial Networkcited by 140
  18. 2016Invertible Conditional GANs for image editingcited by 118
  19. 2020Controlling generative models with continuous factors of variationscited by 93
  20. 2020Improved StyleGAN Embedding: Where are the Good Latents?cited by 88
  21. 2019Image Counterfactual Sensitivity Analysis for Detecting Unintended Biascited by 74
  22. 2020Collaborative Learning for Faster StyleGAN Embeddingcited by 65
  23. 2020GAN "Steerability" without optimizationcited by 48
  24. 2020Improving Inversion and Generation Diversity in StyleGAN using a Gaussianized Latent Spacecited by 34
  25. 2020Style Intervention: How to Achieve Spatial Disentanglement with Style-based Generators?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 160; 2023: 174; 2024: 99; 2025: 45; 2026: 16.

542 papers.

Sort:

By area:

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  8. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  9. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  10. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  11. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  12. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
38 more
  1. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  2. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  3. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  4. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  5. 2021ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinementcited by 248
  6. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  7. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  8. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  9. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  10. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  11. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  12. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  13. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  14. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  15. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  16. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  17. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  18. 2022StyleGAN-Human: A Data-Centric Odyssey of Human Generationcited by 143
  19. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  20. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  21. 2024DF40: Toward Next-Generation Deepfake Detectioncited by 126
  22. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
  23. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  24. 2021FENeRF: Face Editing in Neural Radiance Fieldscited by 112
  25. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  26. 2024Stable Flow: Vital Layers for Training-Free Image Editingcited by 111
  27. 2023Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Modelscited by 98
  28. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  29. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  30. 2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
  31. 2023Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Modelscited by 85
  32. 2021HairCLIP: Design Your Hair by Text and Reference Imagecited by 79
  33. 2023Effective Real Image Editing with Accelerated Iterative Diffusion Inversioncited by 79
  34. 2021CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directionscited by 78
  35. 2023Low-Light Image Enhancement via Structure Modeling and Guidancecited by 78
  36. 2021Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networkscited by 68
  37. 2022Stitch it in Time: GAN-Based Facial Editing of Real Videoscited by 68
  38. 2023StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editingcited by 68

50 most cited of 542 All 542 (JSON)