# Transient Neural Radiance Fields for Lidar View Synthesis and 3D Reconstruction

arXiv: https://arxiv.org/abs/2307.09555 · HTML: https://mlsanity.com/paper/2307.09555/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 69 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [Dense Depth Priors for Neural Radiance Fields from Sparse Input Views](https://mlsanity.com/paper/2112.03288/) (2021)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis](https://mlsanity.com/paper/2109.15271/) (2021)
- [NerfAcc: A General NeRF Acceleration Toolbox](https://mlsanity.com/paper/2210.04847/) (2022)
- [Virtual Wave Optics for Non-Line-of-Sight Imaging](https://mlsanity.com/paper/1810.07535/) (2018)
- [SparsePose: Sparse-View Camera Pose Regression and Refinement](https://mlsanity.com/paper/2211.16991/) (2022)
- [Non-line-of-sight Imaging](https://mlsanity.com/paper/2005.08026/) (2020)
- [Non-line-of-Sight Imaging via Neural Transient Fields](https://mlsanity.com/paper/2101.00373/) (2021)
- [Real-time 3D reconstruction from single-photon lidar data using plug-and-play point cloud denoisers](https://mlsanity.com/paper/1905.06700/) (2019)
- [NeuralBlox: Real-Time Neural Representation Fusion for Robust Volumetric Mapping](https://mlsanity.com/paper/2110.09415/) (2021)
- [The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinement](https://mlsanity.com/paper/2111.13738/) (2021)
- [Seeing Around Corners with Edge-Resolved Transient Imaging](https://mlsanity.com/paper/2002.07118/) (2020)
- [Dead Time Compensation for High-Flux Ranging](https://mlsanity.com/paper/1810.11145/) (2018)
- [High Resolution, Deep Imaging Using Confocal Time-of-flight Diffuse Optical Tomography](https://mlsanity.com/paper/2101.11680/) (2021)

## Built on by (most cited first, 18 of 18)

- [GStex: Per-Primitive Texturing of 2D Gaussian Splatting for Decoupled Appearance and Geometry Modeling](https://mlsanity.com/paper/2409.12954/) (2024)
- [Towards 3D Vision with Low-Cost Single-Photon Cameras](https://mlsanity.com/paper/2403.17801/) (2024)
- [AONeuS: A Neural Rendering Framework for Acoustic-Optical Sensor Fusion](https://mlsanity.com/paper/2402.03309/) (2024)
- [Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB](https://mlsanity.com/paper/2411.19474/) (2024)
- [PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar](https://mlsanity.com/paper/2312.14239/) (2023)
- [Bayesian NeRF: Quantifying Uncertainty with Volume Density for Neural Implicit Fields](https://mlsanity.com/paper/2404.06727/) (2024)
- [Flying with Photons: Rendering Novel Views of Propagating Light](https://mlsanity.com/paper/2404.06493/) (2024)
- [Transientangelo: Few-Viewpoint Surface Reconstruction Using Single-Photon Lidar](https://mlsanity.com/paper/2408.12191/) (2024)
- [SpINR: Neural Volumetric Reconstruction for FMCW Radars](https://mlsanity.com/paper/2503.23313/) (2025)
- [Gated Fields: Learning Scene Reconstruction from Gated Videos](https://mlsanity.com/paper/2405.19819/) (2024)
- [Using a Distance Sensor to Detect Deviations in a Planar Surface](https://mlsanity.com/paper/2408.03838/) (2024)
- [Neural Inverse Rendering from Propagating Light](https://mlsanity.com/paper/2506.05347/) (2025)
- [SpINRv2: Implicit Neural Representation for Passband FMCW Radars](https://mlsanity.com/paper/2506.08163/) (2025)
- [SH-SAS: An Implicit Neural Representation for Complex Spherical-Harmonic Scattering Fields for 3D Synthetic Aperture Sonar](https://mlsanity.com/paper/2509.11087/) (2025)
- [NnD: Diffusion-based Generation of Physically-Nonnegative Objects](https://mlsanity.com/paper/2506.10112/) (2025)
- [SP-TransientBench: A Real-Captured Single Photon Perception Benchmark](https://mlsanity.com/paper/2606.18952/) (2026)
- [Non-line-of-sight imaging with arbitrary relay surface geometries via 3D Gaussian Transient Rendering](https://mlsanity.com/paper/2606.21270/) (2026)
- [Resolving Mixed Single-Photon LiDAR Returns for Foreground-View and Hidden Scene Reconstruction](https://mlsanity.com/paper/2610.01206/) (2026)

## Abstract (arXiv)

Neural radiance fields (NeRFs) have become a ubiquitous tool for modeling scene appearance and geometry from multiview imagery. Recent work has also begun to explore how to use additional supervision from lidar or depth sensor measurements in the NeRF framework. However, previous lidar-supervised NeRFs focus on rendering conventional camera imagery and use lidar-derived point cloud data as auxiliary supervision; thus, they fail to incorporate the underlying image formation model of the lidar. Here, we propose a novel method for rendering transient NeRFs that take as input the raw, time-resolved photon count histograms measured by a single-photon lidar system, and we seek to render such histograms from novel views. Different from conventional NeRFs, the approach relies on a time-resolved version of the volume rendering equation to render the lidar measurements and capture transient light transport phenomena at picosecond timescales. We evaluate our method on a first-of-its-kind dataset of simulated and captured transient multiview scans from a prototype single-photon lidar. Overall, our work brings NeRFs to a new dimension of imaging at transient timescales, newly enabling rendering of transient imagery from novel views. Additionally, we show that our approach recovers improved geometry and conventional appearance compared to point cloud-based supervision when training on few input viewpoints. Transient NeRFs may be especially useful for applications which seek to simulate raw lidar measurements for downstream tasks in autonomous driving, robotics, and remote sensing.