# Flying with Photons: Rendering Novel Views of Propagating Light

arXiv: https://arxiv.org/abs/2404.06493 · HTML: https://mlsanity.com/paper/2404.06493/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 83 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Mixed Neural Voxels for Fast Multi-view Video Synthesis](https://mlsanity.com/paper/2212.00190/) (2022)
- [NerfAcc: Efficient Sampling Accelerates NeRFs](https://mlsanity.com/paper/2305.04966/) (2023)
- [T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis](https://mlsanity.com/paper/2109.15271/) (2021)
- [NerfAcc: A General NeRF Acceleration Toolbox](https://mlsanity.com/paper/2210.04847/) (2022)
- [Non-line-of-sight Imaging](https://mlsanity.com/paper/2005.08026/) (2020)
- [Non-line-of-Sight Imaging via Neural Transient Fields](https://mlsanity.com/paper/2101.00373/) (2021)
- [Neural Microfacet Fields for Inverse Rendering](https://mlsanity.com/paper/2303.17806/) (2023)
- [Eikonal Fields for Refractive Novel-View Synthesis](https://mlsanity.com/paper/2202.00948/) (2022)
- [Nuvo: Neural UV Mapping for Unruly 3D Representations](https://mlsanity.com/paper/2312.05283/) (2023)
- [Transient Neural Radiance Fields for Lidar View Synthesis and 3D Reconstruction](https://mlsanity.com/paper/2307.09555/) (2023)
- [Non-Line-of-Sight Reconstruction using Efficient Transient Rendering](https://mlsanity.com/paper/1809.08044/) (2018)
- [Towards 3D Vision with Low-Cost Single-Photon Cameras](https://mlsanity.com/paper/2403.17801/) (2024)
- [Eclipse: Disambiguating Illumination and Materials using Unintended Shadows](https://mlsanity.com/paper/2305.16321/) (2023)
- [Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective](https://mlsanity.com/paper/1611.00939/) (2016)
- [PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar](https://mlsanity.com/paper/2312.14239/) (2023)
- [Differentiable Transient Rendering](https://mlsanity.com/paper/2206.06193/) (2022)
- [Sampling Neural Radiance Fields for Refractive Objects](https://mlsanity.com/paper/2211.14799/) (2022)
- [Passive Micron-scale Time-of-Flight with Sunlight Interferometry](https://mlsanity.com/paper/2211.10732/) (2022)

## Built on by (most cited first, 7 of 7)

- [Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB](https://mlsanity.com/paper/2411.19474/) (2024)
- [Transientangelo: Few-Viewpoint Surface Reconstruction Using Single-Photon Lidar](https://mlsanity.com/paper/2408.12191/) (2024)
- [Radiance Fields from Photons](https://mlsanity.com/paper/2407.09386/) (2024)
- [Recovering Parametric Scenes from Very Few Time-of-Flight Pixels](https://mlsanity.com/paper/2509.16132/) (2025)
- [Neural Inverse Rendering from Propagating Light](https://mlsanity.com/paper/2506.05347/) (2025)
- [Iterating the Transient Light Transport Matrix for Non-Line-of-Sight Imaging](https://mlsanity.com/paper/2412.10300/) (2024)
- [Resolving Mixed Single-Photon LiDAR Returns for Foreground-View and Hidden Scene Reconstruction](https://mlsanity.com/paper/2610.01206/) (2026)

## Abstract (arXiv)

We present an imaging and neural rendering technique that seeks to synthesize videos of light propagating through a scene from novel, moving camera viewpoints. Our approach relies on a new ultrafast imaging setup to capture a first-of-its kind, multi-viewpoint video dataset with picosecond-level temporal resolution. Combined with this dataset, we introduce an efficient neural volume rendering framework based on the transient field. This field is defined as a mapping from a 3D point and 2D direction to a high-dimensional, discrete-time signal that represents time-varying radiance at ultrafast timescales. Rendering with transient fields naturally accounts for effects due to the finite speed of light, including viewpoint-dependent appearance changes caused by light propagation delays to the camera. We render a range of complex effects, including scattering, specular reflection, refraction, and diffraction. Additionally, we demonstrate removing viewpoint-dependent propagation delays using a time warping procedure, rendering of relativistic effects, and video synthesis of direct and global components of light transport.