# Efficient and Differentiable Shadow Computation for Inverse Problems

arXiv: https://arxiv.org/abs/2104.00359 · HTML: https://mlsanity.com/paper/2104.00359/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [Deep Sky Modeling for Single Image Outdoor Lighting Estimation](https://mlsanity.com/paper/1905.03897/) (2019)
- [Inverse Path Tracing for Joint Material and Lighting Estimation](https://mlsanity.com/paper/1903.07145/) (2019)
- [A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation](https://mlsanity.com/paper/1602.03725/) (2016)
- [Learning View Priors for Single-view 3D Reconstruction](https://mlsanity.com/paper/1811.10719/) (2018)

## Built on by (most cited first, 11 of 11)

- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination](https://mlsanity.com/paper/2207.13607/) (2022)
- [S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Image](https://mlsanity.com/paper/2203.07732/) (2022)
- [HDHumans: A Hybrid Approach for High-fidelity Digital Humans](https://mlsanity.com/paper/2210.12003/) (2022)
- [The Sky's the Limit: Re-lightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibility](https://mlsanity.com/paper/2311.16937/) (2023)
- [Plateau-reduced Differentiable Path Tracing](https://mlsanity.com/paper/2211.17263/) (2022)
- [DeepShadow: Neural Shape from Shadow](https://mlsanity.com/paper/2203.15065/) (2022)
- [Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering](https://mlsanity.com/paper/2310.00362/) (2023)
- [Differentiable Shadow Mapping for Efficient Inverse Graphics](https://mlsanity.com/paper/2308.10896/) (2023)
- [Physics-based Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2212.04705/) (2022)

## Abstract (arXiv)

Differentiable rendering has received increasing interest for image-based inverse problems. It can benefit traditional optimization-based solutions to inverse problems, but also allows for self-supervision of learning-based approaches for which training data with ground truth annotation is hard to obtain. However, existing differentiable renderers either do not model visibility of the light sources from the different points in the scene, responsible for shadows in the images, or are too slow for being used to train deep architectures over thousands of iterations. To this end, we propose an accurate yet efficient approach for differentiable visibility and soft shadow computation. Our approach is based on the spherical harmonics approximations of the scene illumination and visibility, where the occluding surface is approximated with spheres. This allows for a significantly more efficient shadow computation compared to methods based on ray tracing. As our formulation is differentiable, it can be used to solve inverse problems such as texture, illumination, rigid pose, and geometric deformation recovery from images using analysis-by-synthesis optimization.