# PhotonSplat: 3D Scene Reconstruction and Colorization from SPAD Sensors

arXiv: https://arxiv.org/abs/2506.21680 · HTML: https://mlsanity.com/paper/2506.21680/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Builds on (most cited first, 30 of 33)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [Global Structure-from-Motion Revisited](https://mlsanity.com/paper/2407.20219/) (2024)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [COLMAP-Free 3D Gaussian Splatting](https://mlsanity.com/paper/2312.07504/) (2023)
- [4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes](https://mlsanity.com/paper/2402.03307/) (2024)
- [Deblur-NeRF: Neural Radiance Fields from Blurry Images](https://mlsanity.com/paper/2111.14292/) (2021)
- [BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields](https://mlsanity.com/paper/2211.12853/) (2022)
- [Deblurring 3D Gaussian Splatting](https://mlsanity.com/paper/2401.00834/) (2024)
- [NAN: Noise-Aware NeRFs for Burst-Denoising](https://mlsanity.com/paper/2204.04668/) (2022)
- [Lighting up NeRF via Unsupervised Decomposition and Enhancement](https://mlsanity.com/paper/2307.10664/) (2023)
- [DDColor: Towards Photo-Realistic Image Colorization via Dual Decoders](https://mlsanity.com/paper/2212.11613/) (2022)
- [Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption](https://mlsanity.com/paper/2312.09093/) (2023)
- [E2GS: Event Enhanced Gaussian Splatting](https://mlsanity.com/paper/2406.14978/) (2024)
- [Quanta Burst Photography](https://mlsanity.com/paper/2006.11840/) (2020)
- [High Flux Passive Imaging with Single-Photon Sensors](https://mlsanity.com/paper/1902.10190/) (2019)
- [Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion](https://mlsanity.com/paper/2406.02972/) (2024)
- [Thermal-NeRF: Neural Radiance Fields from an Infrared Camera](https://mlsanity.com/paper/2403.10340/) (2024)
- [ThermalNeRF: Thermal Radiance Fields](https://mlsanity.com/paper/2407.15337/) (2024)
- [ThermalGaussian: Thermal 3D Gaussian Splatting](https://mlsanity.com/paper/2409.07200/) (2024)
- [Photon-Starved Scene Inference using Single Photon Cameras](https://mlsanity.com/paper/2107.11001/) (2021)
- [ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization](https://mlsanity.com/paper/2404.06251/) (2024)
- [Megapixel Photon-Counting Color Imaging using Quanta Image Sensor](https://mlsanity.com/paper/1903.09036/) (2019)
- [Radiance Fields from Photons](https://mlsanity.com/paper/2407.09386/) (2024)

## Abstract (arXiv)

Advances in 3D reconstruction using neural rendering have enabled high-quality 3D capture. However, they often fail when the input imagery is corrupted by motion blur, due to fast motion of the camera or the objects in the scene. This work advances neural rendering techniques in such scenarios by using single-photon avalanche diode (SPAD) arrays, an emerging sensing technology capable of sensing images at extremely high speeds. However, the use of SPADs presents its own set of unique challenges in the form of binary images, that are driven by stochastic photon arrivals. To address this, we introduce PhotonSplat, a framework designed to reconstruct 3D scenes directly from SPAD binary images, effectively navigating the noise vs. blur trade-off. Our approach incorporates a novel 3D spatial filtering technique to reduce noise in the renderings. The framework also supports both no-reference using generative priors and reference-based colorization from a single blurry image, enabling downstream applications such as segmentation, object detection and appearance editing tasks. Additionally, we extend our method to incorporate dynamic scene representations, making it suitable for scenes with moving objects. We further contribute PhotonScenes, a real-world multi-view dataset captured with the SPAD sensors.