# Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images using a View-based Representation

arXiv: https://arxiv.org/abs/2003.14166 · HTML: https://mlsanity.com/paper/2003.14166/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2020.

## Built on by (most cited first, 4 of 4)

- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [SSR: Semi-supervised Soft Rasterizer for single-view 2D to 3D Reconstruction](https://mlsanity.com/paper/2108.09593/) (2021)
- [Visual Question Answering From Another Perspective: CLEVR Mental Rotation Tests](https://mlsanity.com/paper/2212.01639/) (2022)
- [Class-Continuous Conditional Generative Neural Radiance Field](https://mlsanity.com/paper/2301.00950/) (2023)

## Abstract (arXiv)

We infer and generate three-dimensional (3D) scene information from a single input image and without supervision. This problem is under-explored, with most prior work relying on supervision from, e.g., 3D ground-truth, multiple images of a scene, image silhouettes or key-points. We propose Pix2Shape, an approach to solve this problem with four components: (i) an encoder that infers the latent 3D representation from an image, (ii) a decoder that generates an explicit 2.5D surfel-based reconstruction of a scene from the latent code (iii) a differentiable renderer that synthesizes a 2D image from the surfel representation, and (iv) a critic network trained to discriminate between images generated by the decoder-renderer and those from a training distribution. Pix2Shape can generate complex 3D scenes that scale with the view-dependent on-screen resolution, unlike representations that capture world-space resolution, i.e., voxels or meshes. We show that Pix2Shape learns a consistent scene representation in its encoded latent space and that the decoder can then be applied to this latent representation in order to synthesize the scene from a novel viewpoint. We evaluate Pix2Shape with experiments on the ShapeNet dataset as well as on a novel benchmark we developed, called 3D-IQTT, to evaluate models based on their ability to enable 3d spatial reasoning. Qualitative and quantitative evaluation demonstrate Pix2Shape's ability to solve scene reconstruction, generation, and understanding tasks.