# PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar

arXiv: https://arxiv.org/abs/2312.14239 · HTML: https://mlsanity.com/paper/2312.14239/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 28 of 28)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis](https://mlsanity.com/paper/2109.15271/) (2021)
- [Neural LiDAR Fields for Novel View Synthesis](https://mlsanity.com/paper/2305.01643/) (2023)
- [LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields](https://mlsanity.com/paper/2304.10406/) (2023)
- [Non-line-of-Sight Imaging via Neural Transient Fields](https://mlsanity.com/paper/2101.00373/) (2021)
- [S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint](https://mlsanity.com/paper/2210.08936/) (2022)
- [Transient Neural Radiance Fields for Lidar View Synthesis and 3D Reconstruction](https://mlsanity.com/paper/2307.09555/) (2023)
- [ShadowNeuS: Neural SDF Reconstruction by Shadow Ray Supervision](https://mlsanity.com/paper/2211.14086/) (2022)
- [Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging](https://mlsanity.com/paper/2205.01679/) (2022)
- [NLOS-NeuS: Non-line-of-sight Neural Implicit Surface](https://mlsanity.com/paper/2303.12280/) (2023)
- [Towards Learning Neural Representations from Shadows](https://mlsanity.com/paper/2203.15946/) (2022)
- [Shadows Shed Light on 3D Objects](https://mlsanity.com/paper/2206.08990/) (2022)
- [DeepShadow: Neural Shape from Shadow](https://mlsanity.com/paper/2203.15065/) (2022)
- [FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras](https://mlsanity.com/paper/2210.02785/) (2022)
- [Role of Transients in Two-Bounce Non-Line-of-Sight Imaging](https://mlsanity.com/paper/2304.01308/) (2023)
- [Depth Fields: Extending Light Field Techniques to Time-of-Flight Imaging](https://mlsanity.com/paper/1509.00816/) (2015)

## Built on by (most cited first, 9 of 9)

- [Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB](https://mlsanity.com/paper/2411.19474/) (2024)
- [Enhancing Autonomous Navigation by Imaging Hidden Objects using Single-Photon LiDAR](https://mlsanity.com/paper/2410.03555/) (2024)
- [Flying with Photons: Rendering Novel Views of Propagating Light](https://mlsanity.com/paper/2404.06493/) (2024)
- [Transientangelo: Few-Viewpoint Surface Reconstruction Using Single-Photon Lidar](https://mlsanity.com/paper/2408.12191/) (2024)
- [Recovering Parametric Scenes from Very Few Time-of-Flight Pixels](https://mlsanity.com/paper/2509.16132/) (2025)
- [Time of the Flight of the Gaussians: Optimizing Depth Indirectly in Dynamic Radiance Fields](https://mlsanity.com/paper/2505.05356/) (2025)
- [Can NeRFs See without Cameras?](https://mlsanity.com/paper/2505.22441/) (2025)
- [Neural Inverse Rendering from Propagating Light](https://mlsanity.com/paper/2506.05347/) (2025)
- [Radar2Shape: 3D Shape Reconstruction from High-Frequency Radar using Multiresolution Signed Distance Functions](https://mlsanity.com/paper/2511.17484/) (2025)

## Abstract (arXiv)

3D reconstruction from a single-view is challenging because of the ambiguity from monocular cues and lack of information about occluded regions. Neural radiance fields (NeRF), while popular for view synthesis and 3D reconstruction, are typically reliant on multi-view images. Existing methods for single-view 3D reconstruction with NeRF rely on either data priors to hallucinate views of occluded regions, which may not be physically accurate, or shadows observed by RGB cameras, which are difficult to detect in ambient light and low albedo backgrounds. We propose using time-of-flight data captured by a single-photon avalanche diode to overcome these limitations. Our method models two-bounce optical paths with NeRF, using lidar transient data for supervision. By leveraging the advantages of both NeRF and two-bounce light measured by lidar, we demonstrate that we can reconstruct visible and occluded geometry without data priors or reliance on controlled ambient lighting or scene albedo. In addition, we demonstrate improved generalization under practical constraints on sensor spatial- and temporal-resolution. We believe our method is a promising direction as single-photon lidars become ubiquitous on consumer devices, such as phones, tablets, and headsets.