# Learning single-image 3D reconstruction by generative modelling of shape, pose and shading

arXiv: https://arxiv.org/abs/1901.06447 · HTML: https://mlsanity.com/paper/1901.06447/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jan 2019.

## Built on by (most cited first, 30 of 43)

- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2312.02201/) (2023)
- [MagicPony: Learning Articulated 3D Animals in the Wild](https://mlsanity.com/paper/2211.12497/) (2022)
- [Stylizing 3D Scene via Implicit Representation and HyperNetwork](https://mlsanity.com/paper/2105.13016/) (2021)
- [Shelf-Supervised Mesh Prediction in the Wild](https://mlsanity.com/paper/2102.06195/) (2021)
- [DOVE: Learning Deformable 3D Objects by Watching Videos](https://mlsanity.com/paper/2107.10844/) (2021)
- [CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields](https://mlsanity.com/paper/2103.17269/) (2021)
- [Learning Generative Models of Textured 3D Meshes from Real-World Images](https://mlsanity.com/paper/2103.15627/) (2021)
- [PartNeRF: Generating Part-Aware Editable 3D Shapes without 3D Supervision](https://mlsanity.com/paper/2303.09554/) (2023)
- [3D-Aware Semantic-Guided Generative Model for Human Synthesis](https://mlsanity.com/paper/2112.01422/) (2021)
- [GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and Consistent 3D Generation](https://mlsanity.com/paper/2311.17971/) (2023)
- [Query2CAD: Generating CAD models using natural language queries](https://mlsanity.com/paper/2406.00144/) (2024)
- [Differentiable Surface Rendering via Non-Differentiable Sampling](https://mlsanity.com/paper/2108.04886/) (2021)
- [PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition](https://mlsanity.com/paper/2203.16670/) (2022)
- [Pre-train, Self-train, Distill: A simple recipe for Supersizing 3D Reconstruction](https://mlsanity.com/paper/2204.03642/) (2022)
- [Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency](https://mlsanity.com/paper/2204.10310/) (2022)
- [CAD: Photorealistic 3D Generation via Adversarial Distillation](https://mlsanity.com/paper/2312.06663/) (2023)
- [3D-Aware Video Generation](https://mlsanity.com/paper/2206.14797/) (2022)
- [ShapeClipper: Scalable 3D Shape Learning from Single-View Images via Geometric and CLIP-based Consistency](https://mlsanity.com/paper/2304.06247/) (2023)
- [DeepMesh: Differentiable Iso-Surface Extraction](https://mlsanity.com/paper/2106.11795/) (2021)
- [Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding](https://mlsanity.com/paper/2403.10395/) (2024)
- [Structural Causal 3D Reconstruction](https://mlsanity.com/paper/2207.10156/) (2022)
- [JointDreamer: Ensuring Geometry Consistency and Text Congruence in Text-to-3D Generation via Joint Score Distillation](https://mlsanity.com/paper/2407.12291/) (2024)
- [CCD-3DR: Consistent Conditioning in Diffusion for Single-Image 3D Reconstruction](https://mlsanity.com/paper/2308.07837/) (2023)
- [Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models](https://mlsanity.com/paper/2406.13099/) (2024)
- [Planes vs. Chairs: Category-guided 3D shape learning without any 3D cues](https://mlsanity.com/paper/2204.10235/) (2022)
- [Creative Birds: Self-Supervised Single-View 3D Style Transfer](https://mlsanity.com/paper/2307.14127/) (2023)
- [Unsupervised Novel View Synthesis from a Single Image](https://mlsanity.com/paper/2102.03285/) (2021)
- [Weakly-Supervised Multi-Face 3D Reconstruction](https://mlsanity.com/paper/2101.02000/) (2021)
- [High-Quality Animatable Dynamic Garment Reconstruction from Monocular Videos](https://mlsanity.com/paper/2311.01214/) (2023)

## Abstract (arXiv)

We present a unified framework tackling two problems: class-specific 3D reconstruction from a single image, and generation of new 3D shape samples. These tasks have received considerable attention recently; however, most existing approaches rely on 3D supervision, annotation of 2D images with keypoints or poses, and/or training with multiple views of each object instance. Our framework is very general: it can be trained in similar settings to existing approaches, while also supporting weaker supervision. Importantly, it can be trained purely from 2D images, without pose annotations, and with only a single view per instance. We employ meshes as an output representation, instead of voxels used in most prior work. This allows us to reason over lighting parameters and exploit shading information during training, which previous 2D-supervised methods cannot. Thus, our method can learn to generate and reconstruct concave object classes. We evaluate our approach in various settings, showing that: (i) it learns to disentangle shape from pose and lighting; (ii) using shading in the loss improves performance compared to just silhouettes; (iii) when using a standard single white light, our model outperforms state-of-the-art 2D-supervised methods, both with and without pose supervision, thanks to exploiting shading cues; (iv) performance improves further when using multiple coloured lights, even approaching that of state-of-the-art 3D-supervised methods; (v) shapes produced by our model capture smooth surfaces and fine details better than voxel-based approaches; and (vi) our approach supports concave classes such as bathtubs and sofas, which methods based on silhouettes cannot learn.