mlsanityData to 7 Oct 2026

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement

arXiv 2104.02699 ↗First posted 6 Apr 2021cs.CVcited by 248

Not on the map. Why?

Cited by 30 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, the power of unconditional image synthesis has significantly advanced through the use of Generative Adversarial Networks (GANs). The task of inverting an image into its corresponding latent code of the trained GAN is of utmost importance as it allows for the manipulation of real images, leveraging the rich semantics learned by the network. Recognizing the limitations of current inversion approaches, in this work we present a novel inversion scheme that extends current encoder-based inversion methods by introducing an iterative refinement mechanism. Instead of directly predicting the latent code of a given real image using a single pass, the encoder is tasked with predicting a residual with respect to the current estimate of the inverted latent code in a self-correcting manner. Our residual-based encoder, named ReStyle, attains improved accuracy compared to current state-of-the-art encoder-based methods with a negligible increase in inference time. We analyze the behavior of ReStyle to gain valuable insights into its iterative nature. We then evaluate the performance of our residual encoder and analyze its robustness compared to optimization-based inversion and state-of-the-art encoders.

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 6 Apr 2021.

Builds on

43 of 60 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  8. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  9. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  10. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  11. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  12. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
31 more
  1. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  2. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  3. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  4. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  5. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  6. 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
  7. 2021GAN Inversion: A Surveycited by 366
  8. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  9. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 352
  10. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  11. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  12. 2020CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognitioncited by 320
  13. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
  14. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 267
  15. 2019On the "steerability" of generative adversarial networkscited by 247
  16. 2019CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learningcited by 217
  17. 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
  18. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  19. 2020Adversarial Latent Autoencoderscited by 155
  20. 2016Inverting The Generator Of A Generative Adversarial Networkcited by 140
  21. 2019Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimationcited by 131
  22. 2015Human Pose Estimation with Iterative Error Feedbackcited by 121
  23. 2019Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesiscited by 106
  24. 2020Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domainscited by 96
  25. 2021Only a Matter of Style: Age Transformation Using a Style-Based Regression Modelcited by 92
  26. 2020Improved StyleGAN Embedding: Where are the Good Latents?cited by 88
  27. 2020Generative Hierarchical Features from Synthesizing Imagescited by 83
  28. 2019Image Counterfactual Sensitivity Analysis for Detecting Unintended Biascited by 74
  29. 2020Collaborative Learning for Faster StyleGAN Embeddingcited by 65
  30. 2021Using latent space regression to analyze and leverage compositionality in GANscited by 55
  31. 2020Face Identity Disentanglement via Latent Space Mappingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 85; 2023: 87; 2024: 31; 2025: 13; 2026: 10.

248 papers.

Sort:

By area:

  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  2. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  3. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  4. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  5. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  6. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  7. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  8. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  9. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  10. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  11. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  12. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
38 more
  1. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  2. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  3. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  4. 2024ReNoise: Real Image Inversion Through Iterative Noisingcited by 127
  5. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
  6. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  7. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  8. 2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
  9. 2023Effective Real Image Editing with Accelerated Iterative Diffusion Inversioncited by 79
  10. 2021CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directionscited by 78
  11. 2022Stitch it in Time: GAN-Based Facial Editing of Real Videoscited by 68
  12. 2022Fine-Grained Face Swapping via Regional GAN Inversioncited by 67
  13. 2022High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimizationcited by 65
  14. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  15. 2024TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Modelscited by 64
  16. 2021GAN-Supervised Dense Visual Alignmentcited by 61
  17. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  18. 2021SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editingcited by 58
  19. 2024Exploiting Style Latent Flows for Generalizing Deepfake Video Detectioncited by 58
  20. 2022Third Time's the Charm? Image and Video Editing with StyleGAN3cited by 54
  21. 2023NOFA: NeRF-based One-shot Facial Avatar Reconstructioncited by 53
  22. 2022GradViT: Gradient Inversion of Vision Transformerscited by 52
  23. 2023StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Videocited by 52
  24. 2025TokenVerse: Versatile Multi-concept Personalization in Token Modulation Spacecited by 51
  25. 2021JoJoGAN: One Shot Face Stylizationcited by 49
  26. 2023HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Facescited by 48
  27. 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Modelscited by 48
  28. 2022InsetGAN for Full-Body Image Generationcited by 45
  29. 2023RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatarscited by 45
  30. 20223D GAN Inversion with Pose Optimizationcited by 44
  31. 2023TriPlaneNet: An Encoder for EG3D Inversioncited by 41
  32. 2021Image-Based CLIP-Guided Essence Transfercited by 39
  33. 2022Spatially-Adaptive Multilayer Selection for GAN Inversion and Editingcited by 39
  34. 2022StyleSwap: Style-Based Generator Empowers Robust Face Swappingcited by 39
  35. 2022Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encodingcited by 39
  36. 20233DAvatarGAN: Bridging Domains for Personalized Editable Avatarscited by 38
  37. 2022Style Transformer for Image Inversion and Editingcited by 37
  38. 2022TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editingcited by 36

50 most cited of 248 All 248 (JSON)