# Make Encoder Great Again in 3D GAN Inversion through Geometry and Occlusion-Aware Encoding

arXiv: https://arxiv.org/abs/2303.12326 · HTML: https://mlsanity.com/paper/2303.12326/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 58 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 30 of 45)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation](https://mlsanity.com/paper/2011.12799/) (2020)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation](https://mlsanity.com/paper/2112.11427/) (2021)
- [ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement](https://mlsanity.com/paper/2104.02699/) (2021)
- [GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation](https://mlsanity.com/paper/2112.08867/) (2021)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [High-Fidelity GAN Inversion for Image Attribute Editing](https://mlsanity.com/paper/2109.06590/) (2021)
- [Editing in Style: Uncovering the Local Semantics of GANs](https://mlsanity.com/paper/2004.14367/) (2020)

## Built on by (most cited first, 30 of 35)

- [Avat3r: Large Animatable Gaussian Reconstruction Model for High-fidelity 3D Head Avatars](https://mlsanity.com/paper/2502.20220/) (2025)
- [TriPlaneNet: An Encoder for EG3D Inversion](https://mlsanity.com/paper/2303.13497/) (2023)
- [VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactment](https://mlsanity.com/paper/2312.04651/) (2023)
- [Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image](https://mlsanity.com/paper/2403.09632/) (2024)
- [DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis](https://mlsanity.com/paper/2312.13016/) (2023)
- [Lite2Relight: 3D-aware Single Image Portrait Relighting](https://mlsanity.com/paper/2407.10487/) (2024)
- [HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillation](https://mlsanity.com/paper/2312.07539/) (2023)
- [FaceLift: Learning Generalizable Single Image 3D Face Reconstruction from Synthetic Heads](https://mlsanity.com/paper/2412.17812/) (2024)
- [Generalizable One-shot Neural Head Avatar](https://mlsanity.com/paper/2306.08768/) (2023)
- [Real-time 3D-aware Portrait Video Relighting](https://mlsanity.com/paper/2410.18355/) (2024)
- [Generative Unlearning for Any Identity](https://mlsanity.com/paper/2405.09879/) (2024)
- [Learning to Generate Conditional Tri-plane for 3D-aware Expression Controllable Portrait Animation](https://mlsanity.com/paper/2404.00636/) (2024)
- [Reference-Based 3D-Aware Image Editing with Triplanes](https://mlsanity.com/paper/2404.03632/) (2024)
- [ESC: Erasing Space Concept for Knowledge Deletion](https://mlsanity.com/paper/2504.02199/) (2025)
- [My3DGen: A Scalable Personalized 3D Generative Model](https://mlsanity.com/paper/2307.05468/) (2023)
- [PanoLAM: Large Avatar Model for Gaussian Full-Head Synthesis from One-shot Unposed Image](https://mlsanity.com/paper/2509.07552/) (2025)
- [Warping the Residuals for Image Editing with StyleGAN](https://mlsanity.com/paper/2312.11422/) (2023)
- [DiffAge3D: Diffusion-based 3D-aware Face Aging](https://mlsanity.com/paper/2408.15922/) (2024)
- [Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images](https://mlsanity.com/paper/2409.20530/) (2024)
- [G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images](https://mlsanity.com/paper/2404.07474/) (2024)
- [FED-NeRF: Achieve High 3D Consistency and Temporal Coherence for Face Video Editing on Dynamic NeRF](https://mlsanity.com/paper/2401.02616/) (2024)
- [Talk3D: High-Fidelity Talking Portrait Synthesis via Personalized 3D Generative Prior](https://mlsanity.com/paper/2403.20153/) (2024)
- [Revealing Directions for Text-guided 3D Face Editing](https://mlsanity.com/paper/2410.04965/) (2024)
- [Identity Preserving 3D Head Stylization with Multiview Score Distillation](https://mlsanity.com/paper/2411.13536/) (2024)
- [FlexAvatar: Flexible Large Reconstruction Model for Animatable Gaussian Head Avatars with Detailed Deformation](https://mlsanity.com/paper/2512.17717/) (2025)
- [SUPER: Selfie Undistortion and Head Pose Editing with Identity Preservation](https://mlsanity.com/paper/2406.12700/) (2024)
- [SUGAR: A Sweeter Spot for Generative Unlearning of Many Identities](https://mlsanity.com/paper/2512.06562/) (2025)
- [From Blurry to Believable: Enhancing Low-quality Talking Heads with 3D Generative Priors](https://mlsanity.com/paper/2602.06122/) (2026)
- [Large-Scale High-Quality 3D Gaussian Head Reconstruction from Multi-View Captures](https://mlsanity.com/paper/2605.04035/) (2026)
- [A Reference-Based 3D Semantic-Aware Framework for Accurate Local Facial Attribute Editing](https://mlsanity.com/paper/2407.18392/) (2024)

## Abstract (arXiv)

3D GAN inversion aims to achieve high reconstruction fidelity and reasonable 3D geometry simultaneously from a single image input. However, existing 3D GAN inversion methods rely on time-consuming optimization for each individual case. In this work, we introduce a novel encoder-based inversion framework based on EG3D, one of the most widely-used 3D GAN models. We leverage the inherent properties of EG3D's latent space to design a discriminator and a background depth regularization. This enables us to train a geometry-aware encoder capable of converting the input image into corresponding latent code. Additionally, we explore the feature space of EG3D and develop an adaptive refinement stage that improves the representation ability of features in EG3D to enhance the recovery of fine-grained textural details. Finally, we propose an occlusion-aware fusion operation to prevent distortion in unobserved regions. Our method achieves impressive results comparable to optimization-based methods while operating up to 500 times faster. Our framework is well-suited for applications such as semantic editing.