# Shape from Shading for Robotic Manipulation

arXiv: https://arxiv.org/abs/2304.11824 · HTML: https://mlsanity.com/paper/2304.11824/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Revisiting Active Perception](https://mlsanity.com/paper/1603.02729/) (2016)
- [NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields](https://mlsanity.com/paper/2203.01913/) (2022)
- [Intel RealSense Stereoscopic Depth Cameras](https://mlsanity.com/paper/1705.05548/) (2017)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo](https://mlsanity.com/paper/2207.11406/) (2022)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint](https://mlsanity.com/paper/2210.08936/) (2022)
- [PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition](https://mlsanity.com/paper/2203.16670/) (2022)
- [VIRDO: Visio-tactile Implicit Representations of Deformable Objects](https://mlsanity.com/paper/2202.00868/) (2022)
- [WildLight: In-the-wild Inverse Rendering with a Flashlight](https://mlsanity.com/paper/2303.14190/) (2023)
- [CNN based Learning using Reflection and Retinex Models for Intrinsic Image Decomposition](https://mlsanity.com/paper/1712.01056/) (2017)
- [Intrinsic Image Decomposition using Paradigms](https://mlsanity.com/paper/2011.10512/) (2020)
- [VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects](https://mlsanity.com/paper/2210.03701/) (2022)
- [From Shading to Local Shape](https://mlsanity.com/paper/1310.2916/) (2013)
- [Towards Learning Neural Representations from Shadows](https://mlsanity.com/paper/2203.15946/) (2022)
- [Using Collocated Vision and Tactile Sensors for Visual Servoing and Localization](https://mlsanity.com/paper/2204.11686/) (2022)

## Abstract (arXiv)

Controlling illumination can generate high quality information about object surface normals and depth discontinuities at a low computational cost. In this work we demonstrate a robot workspace-scaled controlled illumination approach that generates high quality information for table top scale objects for robotic manipulation. With our low angle of incidence directional illumination approach, we can precisely capture surface normals and depth discontinuities of monochromatic Lambertian objects. We show that this approach to shape estimation is 1) valuable for general purpose grasping with a single point vacuum gripper, 2) can measure the deformation of known objects, and 3) can estimate pose of known objects and track unknown objects in the robot's workspace.