# Self-Calibrating, Fully Differentiable NLOS Inverse Rendering

arXiv: https://arxiv.org/abs/2309.12047 · HTML: https://mlsanity.com/paper/2309.12047/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 49 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [Virtual Wave Optics for Non-Line-of-Sight Imaging](https://mlsanity.com/paper/1810.07535/) (2018)
- [Non-line-of-sight Imaging](https://mlsanity.com/paper/2005.08026/) (2020)
- [Non-line-of-Sight Imaging via Neural Transient Fields](https://mlsanity.com/paper/2101.00373/) (2021)
- [Fast Back-Projection for Non-Line of Sight Reconstruction](https://mlsanity.com/paper/1703.02016/) (2017)
- [Deep Non-Line-of-Sight Reconstruction](https://mlsanity.com/paper/2001.09067/) (2020)
- [Real-time Non-line-of-Sight imaging of dynamic scenes](https://mlsanity.com/paper/2010.12737/) (2020)
- [Non-Line-of-Sight Reconstruction using Efficient Transient Rendering](https://mlsanity.com/paper/1809.08044/) (2018)
- [Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging](https://mlsanity.com/paper/2205.01679/) (2022)
- [A Computational Model of a Single-Photon Avalanche Diode Sensor for Transient Imaging](https://mlsanity.com/paper/1703.02635/) (2017)
- [NLOS-NeuS: Non-line-of-sight Neural Implicit Surface](https://mlsanity.com/paper/2303.12280/) (2023)
- [Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective](https://mlsanity.com/paper/1611.00939/) (2016)
- [Differentiable Transient Rendering](https://mlsanity.com/paper/2206.06193/) (2022)
- [Virtual Light Transport Matrices for Non-Line-Of-Sight Imaging](https://mlsanity.com/paper/2103.12622/) (2021)

## Built on by (most cited first, 3 of 3)

- [MARMOT: Masked Autoencoder for Modeling Transient Imaging](https://mlsanity.com/paper/2506.08470/) (2025)
- [Non-line-of-sight imaging with arbitrary relay surface geometries via 3D Gaussian Transient Rendering](https://mlsanity.com/paper/2606.21270/) (2026)
- [Cascaded Non-Line-of-Sight Imaging](https://mlsanity.com/paper/2609.16017/) (2026)

## Abstract (arXiv)

Existing time-resolved non-line-of-sight (NLOS) imaging methods reconstruct hidden scenes by inverting the optical paths of indirect illumination measured at visible relay surfaces. These methods are prone to reconstruction artifacts due to inversion ambiguities and capture noise, which are typically mitigated through the manual selection of filtering functions and parameters. We introduce a fully-differentiable end-to-end NLOS inverse rendering pipeline that self-calibrates the imaging parameters during the reconstruction of hidden scenes, using as input only the measured illumination while working both in the time and frequency domains. Our pipeline extracts a geometric representation of the hidden scene from NLOS volumetric intensities and estimates the time-resolved illumination at the relay wall produced by such geometric information using differentiable transient rendering. We then use gradient descent to optimize imaging parameters by minimizing the error between our simulated time-resolved illumination and the measured illumination. Our end-to-end differentiable pipeline couples diffraction-based volumetric NLOS reconstruction with path-space light transport and a simple ray marching technique to extract detailed, dense sets of surface points and normals of hidden scenes. We demonstrate the robustness of our method to consistently reconstruct geometry and albedo, even under significant noise levels.