# Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging

arXiv: https://arxiv.org/abs/2205.01679 · HTML: https://mlsanity.com/paper/2205.01679/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Editing Conditional Radiance Fields](https://mlsanity.com/paper/2105.06466/) (2021)
- [T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis](https://mlsanity.com/paper/2109.15271/) (2021)
- [Virtual Wave Optics for Non-Line-of-Sight Imaging](https://mlsanity.com/paper/1810.07535/) (2018)
- [Non-line-of-sight Imaging](https://mlsanity.com/paper/2005.08026/) (2020)
- [Non-line-of-Sight Imaging via Neural Transient Fields](https://mlsanity.com/paper/2101.00373/) (2021)
- [Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar](https://mlsanity.com/paper/1912.06613/) (2019)
- [Optical Non-Line-of-Sight Physics-based 3D Human Pose Estimation](https://mlsanity.com/paper/2003.14414/) (2020)
- [Fast Back-Projection for Non-Line of Sight Reconstruction](https://mlsanity.com/paper/1703.02016/) (2017)
- [Deep Non-Line-of-Sight Reconstruction](https://mlsanity.com/paper/2001.09067/) (2020)
- [Real-time Non-line-of-Sight imaging of dynamic scenes](https://mlsanity.com/paper/2010.12737/) (2020)
- [Non-Line-of-Sight Reconstruction using Efficient Transient Rendering](https://mlsanity.com/paper/1809.08044/) (2018)
- [A Computational Model of a Single-Photon Avalanche Diode Sensor for Transient Imaging](https://mlsanity.com/paper/1703.02635/) (2017)
- [Virtual Light Transport Matrices for Non-Line-Of-Sight Imaging](https://mlsanity.com/paper/2103.12622/) (2021)

## Built on by (most cited first, 15 of 15)

- [Towards 3D Vision with Low-Cost Single-Photon Cameras](https://mlsanity.com/paper/2403.17801/) (2024)
- [NLOS-NeuS: Non-line-of-sight Neural Implicit Surface](https://mlsanity.com/paper/2303.12280/) (2023)
- [Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB](https://mlsanity.com/paper/2411.19474/) (2024)
- [PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar](https://mlsanity.com/paper/2312.14239/) (2023)
- [Unlocking the Performance of Proximity Sensors by Utilizing Transient Histograms](https://mlsanity.com/paper/2308.13473/) (2023)
- [Omni-Line-of-Sight Imaging for Holistic Shape Reconstruction](https://mlsanity.com/paper/2304.10780/) (2023)
- [Self-Calibrating, Fully Differentiable NLOS Inverse Rendering](https://mlsanity.com/paper/2309.12047/) (2023)
- [Learning to Enhance Aperture Phasor Field for Non-Line-of-Sight Imaging](https://mlsanity.com/paper/2407.18574/) (2024)
- [Generalizable Non-Line-of-Sight Imaging with Learnable Physical Priors](https://mlsanity.com/paper/2409.14011/) (2024)
- [Domain Reduction Strategy for Non Line of Sight Imaging](https://mlsanity.com/paper/2308.10269/) (2023)
- [MARMOT: Masked Autoencoder for Modeling Transient Imaging](https://mlsanity.com/paper/2506.08470/) (2025)
- [TransiT: Transient Transformer for Non-line-of-sight Videography](https://mlsanity.com/paper/2503.11328/) (2025)
- [Fast and Memory-efficient Non-line-of-sight Imaging with Quasi-Fresnel Transform](https://mlsanity.com/paper/2508.02003/) (2025)
- [3D Reconstruction from Transient Measurements with Time-Resolved Transformer](https://mlsanity.com/paper/2510.09205/) (2025)
- [Memory-efficient GPU pipelines for real-time non-line-of-sight reconstruction](https://mlsanity.com/paper/2608.28183/) (2026)

## Abstract (arXiv)

Computational approach to imaging around the corner, or non-line-of-sight (NLOS) imaging, is becoming a reality thanks to major advances in imaging hardware and reconstruction algorithms. A recent development towards practical NLOS imaging, Nam et al. demonstrated a high-speed non-confocal imaging system that operates at 5Hz, 100x faster than the prior art. This enormous gain in acquisition rate, however, necessitates numerous approximations in light transport, breaking many existing NLOS reconstruction methods that assume an idealized image formation model. To bridge the gap, we present a novel deep model that incorporates the complementary physics priors of wave propagation and volume rendering into a neural network for high-quality and robust NLOS reconstruction. This orchestrated design regularizes the solution space by relaxing the image formation model, resulting in a deep model that generalizes well on real captures despite being exclusively trained on synthetic data. Further, we devise a unified learning framework that enables our model to be flexibly trained using diverse supervision signals, including target intensity images or even raw NLOS transient measurements. Once trained, our model renders both intensity and depth images at inference time in a single forward pass, capable of processing more than 5 captures per second on a high-end GPU. Through extensive qualitative and quantitative experiments, we show that our method outperforms prior physics and learning based approaches on both synthetic and real measurements. We anticipate that our method along with the fast capturing system will accelerate future development of NLOS imaging for real world applications that require high-speed imaging.