# Shadows Shed Light on 3D Objects

arXiv: https://arxiv.org/abs/2206.08990 · HTML: https://mlsanity.com/paper/2206.08990/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 22 of 22)

- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models](https://mlsanity.com/paper/2003.03808/) (2020)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [Deep Learning Techniques for Inverse Problems in Imaging](https://mlsanity.com/paper/2005.06001/) (2020)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [Generative and Discriminative Voxel Modeling with Convolutional Neural Networks](https://mlsanity.com/paper/1608.04236/) (2016)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [Shelf-Supervised Mesh Prediction in the Wild](https://mlsanity.com/paper/2102.06195/) (2021)
- [Instance Shadow Detection](https://mlsanity.com/paper/1911.07034/) (2019)
- [Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images](https://mlsanity.com/paper/1508.04546/) (2015)
- [Image2Mesh: A Learning Framework for Single Image 3D Reconstruction](https://mlsanity.com/paper/1711.10669/) (2017)
- [Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction](https://mlsanity.com/paper/1804.06032/) (2018)
- [Shadow Art Revisited: A Differentiable Rendering Based Approach](https://mlsanity.com/paper/2107.14539/) (2021)

## Built on by (most cited first, 8 of 8)

- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [pix2gestalt: Amodal Segmentation by Synthesizing Wholes](https://mlsanity.com/paper/2401.14398/) (2024)
- [FLEX: Full-Body Grasping Without Full-Body Grasps](https://mlsanity.com/paper/2211.11903/) (2022)
- [Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape](https://mlsanity.com/paper/2305.15399/) (2023)
- [Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection](https://mlsanity.com/paper/2305.01652/) (2023)
- [PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar](https://mlsanity.com/paper/2312.14239/) (2023)
- [Probabilistic Shape Completion by Estimating Canonical Factors with Hierarchical VAE](https://mlsanity.com/paper/2212.03370/) (2022)

## Abstract (arXiv)

3D reconstruction is a fundamental problem in computer vision, and the task is especially challenging when the object to reconstruct is partially or fully occluded. We introduce a method that uses the shadows cast by an unobserved object in order to infer the possible 3D volumes behind the occlusion. We create a differentiable image formation model that allows us to jointly infer the 3D shape of an object, its pose, and the position of a light source. Since the approach is end-to-end differentiable, we are able to integrate learned priors of object geometry in order to generate realistic 3D shapes of different object categories. Experiments and visualizations show that the method is able to generate multiple possible solutions that are consistent with the observation of the shadow. Our approach works even when the position of the light source and object pose are both unknown. Our approach is also robust to real-world images where ground-truth shadow mask is unknown.