# Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

arXiv: https://arxiv.org/abs/2011.00844 · HTML: https://mlsanity.com/paper/2011.00844/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2020.

## Built on by (most cited first, 30 of 63)

- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023)
- [CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis](https://mlsanity.com/paper/2110.09788/) (2021)
- [EVA3D: Compositional 3D Human Generation from 2D Image Collections](https://mlsanity.com/paper/2210.04888/) (2022)
- [EpiGRAF: Rethinking training of 3D GANs](https://mlsanity.com/paper/2206.10535/) (2022)
- [Talk-to-Edit: Fine-Grained Facial Editing via Dialog](https://mlsanity.com/paper/2109.04425/) (2021)
- [GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds](https://mlsanity.com/paper/2104.07659/) (2021)
- [LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation](https://mlsanity.com/paper/2403.12019/) (2024)
- [A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis](https://mlsanity.com/paper/2110.15678/) (2021)
- [Real-Time Radiance Fields for Single-Image Portrait View Synthesis](https://mlsanity.com/paper/2305.02310/) (2023)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [DOVE: Learning Deformable 3D Objects by Watching Videos](https://mlsanity.com/paper/2107.10844/) (2021)
- [Labels4Free: Unsupervised Segmentation using StyleGAN](https://mlsanity.com/paper/2103.14968/) (2021)
- [Ensembling with Deep Generative Views](https://mlsanity.com/paper/2104.14551/) (2021)
- [Generative Occupancy Fields for 3D Surface-Aware Image Synthesis](https://mlsanity.com/paper/2111.00969/) (2021)
- [Generative Models: What Do They Know? Do They Know Things? Let's Find Out!](https://mlsanity.com/paper/2311.17137/) (2023)
- [Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models](https://mlsanity.com/paper/2105.08127/) (2021)
- [StyleGAN knows Normal, Depth, Albedo, and More](https://mlsanity.com/paper/2306.00987/) (2023)
- [AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis](https://mlsanity.com/paper/2204.03105/) (2022)
- [3DAvatarGAN: Bridging Domains for Personalized Editable Avatars](https://mlsanity.com/paper/2301.02700/) (2023)
- [REALY: Rethinking the Evaluation of 3D Face Reconstruction](https://mlsanity.com/paper/2203.09729/) (2022)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [Towards Open-World Text-Guided Face Image Generation and Manipulation](https://mlsanity.com/paper/2104.08910/) (2021)
- [Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views](https://mlsanity.com/paper/2308.14078/) (2023)
- [MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation](https://mlsanity.com/paper/2111.01048/) (2021)
- [Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion](https://mlsanity.com/paper/2212.07409/) (2022)
- [3DFaceShop: Explicitly Controllable 3D-Aware Portrait Generation](https://mlsanity.com/paper/2209.05434/) (2022)
- [Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing](https://mlsanity.com/paper/2312.03763/) (2023)
- [CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images](https://mlsanity.com/paper/2308.12288/) (2023)
- [Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation](https://mlsanity.com/paper/2303.09036/) (2023)

## Abstract (arXiv)

Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.