# Im2Avatar: Colorful 3D Reconstruction from a Single Image

arXiv: https://arxiv.org/abs/1804.06375 · HTML: https://mlsanity.com/paper/1804.06375/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2018.

## Built on by (most cited first, 8 of 8)

- [Texturify: Generating Textures on 3D Shape Surfaces](https://mlsanity.com/paper/2204.02411/) (2022)
- [Texture Generation on 3D Meshes with Point-UV Diffusion](https://mlsanity.com/paper/2308.10490/) (2023)
- [S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling](https://mlsanity.com/paper/2101.06571/) (2021)
- [Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation](https://mlsanity.com/paper/2204.10850/) (2022)
- [AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis](https://mlsanity.com/paper/2204.03105/) (2022)
- [Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction](https://mlsanity.com/paper/2104.00858/) (2021)
- [Deep Hybrid Self-Prior for Full 3D Mesh Generation](https://mlsanity.com/paper/2108.08017/) (2021)
- [Free-form 3D Scene Inpainting with Dual-stream GAN](https://mlsanity.com/paper/2212.08464/) (2022)

## Abstract (arXiv)

Existing works on single-image 3D reconstruction mainly focus on shape recovery. In this work, we study a new problem, that is, simultaneously recovering 3D shape and surface color from a single image, namely "colorful 3D reconstruction". This problem is both challenging and intriguing because the ability to infer textured 3D model from a single image is at the core of visual understanding. Here, we propose an end-to-end trainable framework, Colorful Voxel Network (CVN), to tackle this problem. Conditioned on a single 2D input, CVN learns to decompose shape and surface color information of a 3D object into a 3D shape branch and a surface color branch, respectively. Specifically, for the shape recovery, we generate a shape volume with the state of its voxels indicating occupancy. For the surface color recovery, we combine the strength of appearance hallucination and geometric projection by concurrently learning a regressed color volume and a 2D-to-3D flow volume, which are then fused into a blended color volume. The final textured 3D model is obtained by sampling color from the blended color volume at the positions of occupied voxels in the shape volume. To handle the severe sparse volume representations, a novel loss function, Mean Squared False Cross-Entropy Loss (MSFCEL), is designed. Extensive experiments demonstrate that our approach achieves significant improvement over baselines, and shows great generalization across diverse object categories and arbitrary viewpoints.