# Progressive Learning of 3D Reconstruction Network from 2D GAN Data

arXiv: https://arxiv.org/abs/2305.11102 · HTML: https://mlsanity.com/paper/2305.11102/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 63 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 30 of 53)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [High-Fidelity GAN Inversion for Image Attribute Editing](https://mlsanity.com/paper/2109.06590/) (2021)

## Built on by (most cited first, 10 of 10)

- [Adaptive Shells for Efficient Neural Radiance Field Rendering](https://mlsanity.com/paper/2311.10091/) (2023)
- [Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes](https://mlsanity.com/paper/2312.04043/) (2023)
- [3D Stylization via Large Reconstruction Model](https://mlsanity.com/paper/2504.21836/) (2025)
- [Warping the Residuals for Image Editing with StyleGAN](https://mlsanity.com/paper/2312.11422/) (2023)
- [NOVA-3D: Non-overlapped Views for 3D Anime Character Reconstruction](https://mlsanity.com/paper/2405.12505/) (2024)
- [Identity Preserving 3D Head Stylization with Multiview Score Distillation](https://mlsanity.com/paper/2411.13536/) (2024)
- [MD-ProjTex: Texturing 3D Shapes with Multi-Diffusion Projection](https://mlsanity.com/paper/2504.02762/) (2025)
- [Diverse Semantic Image Editing with Style Codes](https://mlsanity.com/paper/2309.13975/) (2023)
- [Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusion](https://mlsanity.com/paper/2411.10369/) (2024)
- [GeoFusionLRM: Geometry-Aware Self-Correction for Consistent 3D Reconstruction](https://mlsanity.com/paper/2602.14119/) (2026)

## Abstract (arXiv)

This paper presents a method to reconstruct high-quality textured 3D models from single images. Current methods rely on datasets with expensive annotations; multi-view images and their camera parameters. Our method relies on GAN generated multi-view image datasets which have a negligible annotation cost. However, they are not strictly multi-view consistent and sometimes GANs output distorted images. This results in degraded reconstruction qualities. In this work, to overcome these limitations of generated datasets, we have two main contributions which lead us to achieve state-of-the-art results on challenging objects: 1) A robust multi-stage learning scheme that gradually relies more on the models own predictions when calculating losses, 2) A novel adversarial learning pipeline with online pseudo-ground truth generations to achieve fine details. Our work provides a bridge from 2D supervisions of GAN models to 3D reconstruction models and removes the expensive annotation efforts. We show significant improvements over previous methods whether they were trained on GAN generated multi-view images or on real images with expensive annotations. Please visit our web-page for 3D visuals: https://research.nvidia.com/labs/adlr/progressive-3d-learning