# Unsupervised Learning of 3D Structure from Images

arXiv: https://arxiv.org/abs/1607.00662 · HTML: https://mlsanity.com/paper/1607.00662/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2016.

## Built on by (most cited first, 30 of 67)

- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation](https://mlsanity.com/paper/2112.11427/) (2021)
- [MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers](https://mlsanity.com/paper/2311.15475/) (2023)
- [3D Neural Scene Representations for Visuomotor Control](https://mlsanity.com/paper/2107.04004/) (2021)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Unsupervised Learning of 3D Object Categories from Videos in the Wild](https://mlsanity.com/paper/2103.16552/) (2021)
- [Deep Learning on Implicit Neural Representations of Shapes](https://mlsanity.com/paper/2302.05438/) (2023)
- [ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model](https://mlsanity.com/paper/2207.09446/) (2022)
- [CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields](https://mlsanity.com/paper/2103.17269/) (2021)
- [GIFS: Neural Implicit Function for General Shape Representation](https://mlsanity.com/paper/2204.07126/) (2022)
- [Neural Volumetric Memory for Visual Locomotion Control](https://mlsanity.com/paper/2304.01201/) (2023)
- [Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey](https://mlsanity.com/paper/2411.03688/) (2024)
- [Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views](https://mlsanity.com/paper/2111.07117/) (2021)
- [Video Autoencoder: self-supervised disentanglement of static 3D structure and motion](https://mlsanity.com/paper/2110.02951/) (2021)
- [3D-Aware Semantic-Guided Generative Model for Human Synthesis](https://mlsanity.com/paper/2112.01422/) (2021)
- [DoF-NeRF: Depth-of-Field Meets Neural Radiance Fields](https://mlsanity.com/paper/2208.00945/) (2022)
- [GINA-3D: Learning to Generate Implicit Neural Assets in the Wild](https://mlsanity.com/paper/2304.02163/) (2023)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [ALTO: Alternating Latent Topologies for Implicit 3D Reconstruction](https://mlsanity.com/paper/2212.04096/) (2022)
- [SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images](https://mlsanity.com/paper/2212.09100/) (2022)
- [Decoupling Dynamic Monocular Videos for Dynamic View Synthesis](https://mlsanity.com/paper/2304.01716/) (2023)
- [NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Dropping](https://mlsanity.com/paper/2111.15207/) (2021)
- [Neural Vector Fields: Implicit Representation by Explicit Learning](https://mlsanity.com/paper/2303.04341/) (2023)
- [Primitive-based Shape Abstraction via Nonparametric Bayesian Inference](https://mlsanity.com/paper/2203.14714/) (2022)
- [PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers](https://mlsanity.com/paper/2407.13677/) (2024)
- [DeepMesh: Differentiable Iso-Surface Extraction](https://mlsanity.com/paper/2106.11795/) (2021)
- [Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation](https://mlsanity.com/paper/2307.03869/) (2023)

## Abstract (arXiv)

A key goal of computer vision is to recover the underlying 3D structure from 2D observations of the world. In this paper we learn strong deep generative models of 3D structures, and recover these structures from 3D and 2D images via probabilistic inference. We demonstrate high-quality samples and report log-likelihoods on several datasets, including ShapeNet [2], and establish the first benchmarks in the literature. We also show how these models and their inference networks can be trained end-to-end from 2D images. This demonstrates for the first time the feasibility of learning to infer 3D representations of the world in a purely unsupervised manner.