# Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion

arXiv: https://arxiv.org/abs/2212.07409 · HTML: https://mlsanity.com/paper/2212.07409/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 71 references that are in the map.
- First posted on arXiv 14 Dec 2022.

## Builds on (most cited first, 30 of 61)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)

## Built on by (most cited first, 24 of 24)

- [LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation](https://mlsanity.com/paper/2403.12019/) (2024)
- [TriPlaneNet: An Encoder for EG3D Inversion](https://mlsanity.com/paper/2303.13497/) (2023)
- [Make Encoder Great Again in 3D GAN Inversion through Geometry and Occlusion-Aware Encoding](https://mlsanity.com/paper/2303.12326/) (2023)
- [Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing](https://mlsanity.com/paper/2312.03763/) (2023)
- [DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors](https://mlsanity.com/paper/2312.16837/) (2023)
- [One2Avatar: Generative Implicit Head Avatar For Few-shot User Adaptation](https://mlsanity.com/paper/2402.11909/) (2024)
- [Learning Dense Correspondence for NeRF-Based Face Reenactment](https://mlsanity.com/paper/2312.10422/) (2023)
- [InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image](https://mlsanity.com/paper/2311.02826/) (2023)
- [MVDD: Multi-View Depth Diffusion Models](https://mlsanity.com/paper/2312.04875/) (2023)
- [DeformToon3D: Deformable 3D Toonification from Neural Radiance Fields](https://mlsanity.com/paper/2309.04410/) (2023)
- [Reference-Based 3D-Aware Image Editing with Triplanes](https://mlsanity.com/paper/2404.03632/) (2024)
- [Correspondence Distillation from NeRF-based GAN](https://mlsanity.com/paper/2212.09735/) (2022)
- [WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space](https://mlsanity.com/paper/2311.13570/) (2023)
- [Warping the Residuals for Image Editing with StyleGAN](https://mlsanity.com/paper/2312.11422/) (2023)
- [PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editing](https://mlsanity.com/paper/2511.02777/) (2025)
- [In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing](https://mlsanity.com/paper/2302.04871/) (2023)
- [3D-GOI: 3D GAN Omni-Inversion for Multifaceted and Multi-object Editing](https://mlsanity.com/paper/2311.12050/) (2023)
- [INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned Hypernetworks with Neural Radiance Fields](https://mlsanity.com/paper/2312.16197/) (2023)
- [G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images](https://mlsanity.com/paper/2404.07474/) (2024)
- [Refining 3D Human Texture Estimation from a Single Image](https://mlsanity.com/paper/2303.03471/) (2023)
- [ToonifyGB: StyleGAN-based Gaussian Blendshapes for 3D Stylized Head Avatars](https://mlsanity.com/paper/2505.10072/) (2025)
- [Self-supervised Learning for Enhancing Geometrical Modeling in 3D-Aware Generative Adversarial Network](https://mlsanity.com/paper/2312.11856/) (2023)
- [FFaceNeRF: Few-shot Face Editing in Neural Radiance Fields](https://mlsanity.com/paper/2503.17095/) (2025)
- [Survey on Controlable Image Synthesis with Deep Learning](https://mlsanity.com/paper/2307.10275/) (2023)

## Abstract (arXiv)

StyleGAN has achieved great progress in 2D face reconstruction and semantic editing via image inversion and latent editing. While studies over extending 2D StyleGAN to 3D faces have emerged, a corresponding generic 3D GAN inversion framework is still missing, limiting the applications of 3D face reconstruction and semantic editing. In this paper, we study the challenging problem of 3D GAN inversion where a latent code is predicted given a single face image to faithfully recover its 3D shapes and detailed textures. The problem is ill-posed: innumerable compositions of shape and texture could be rendered to the current image. Furthermore, with the limited capacity of a global latent code, 2D inversion methods cannot preserve faithful shape and texture at the same time when applied to 3D models. To solve this problem, we devise an effective self-training scheme to constrain the learning of inversion. The learning is done efficiently without any real-world 2D-3D training pairs but proxy samples generated from a 3D GAN. In addition, apart from a global latent code that captures the coarse shape and texture information, we augment the generation network with a local branch, where pixel-aligned features are added to faithfully reconstruct face details. We further consider a new pipeline to perform 3D view-consistent editing. Extensive experiments show that our method outperforms state-of-the-art inversion methods in both shape and texture reconstruction quality. Code and data will be released.