# 3D Scene Inference from Transient Histograms

arXiv: https://arxiv.org/abs/2211.05094 · HTML: https://mlsanity.com/paper/2211.05094/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Builds on (most cited first, 21 of 21)

- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [High Quality Monocular Depth Estimation via Transfer Learning](https://mlsanity.com/paper/1812.11941/) (2018)
- [Domain-invariant Stereo Matching Networks](https://mlsanity.com/paper/1911.13287/) (2019)
- [PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image](https://mlsanity.com/paper/1812.04072/) (2018)
- [Guiding Monocular Depth Estimation Using Depth-Attention Volume](https://mlsanity.com/paper/2004.02760/) (2020)
- [PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1804.06278/) (2018)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Deformable Kernel Networks for Joint Image Filtering](https://mlsanity.com/paper/1910.08373/) (2019)
- [Bidirectional Attention Network for Monocular Depth Estimation](https://mlsanity.com/paper/2009.00743/) (2020)
- [Detail Preserving Depth Estimation from a Single Image Using Attention Guided Networks](https://mlsanity.com/paper/1809.00646/) (2018)
- [Asynchronous Single-Photon 3D Imaging](https://mlsanity.com/paper/1908.06372/) (2019)
- [Generating and Exploiting Probabilistic Monocular Depth Estimates](https://mlsanity.com/paper/1906.05739/) (2019)
- [Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path](https://mlsanity.com/paper/1912.06727/) (2019)
- [Depth Sensing Beyond LiDAR Range](https://mlsanity.com/paper/2004.03048/) (2020)

## Built on by (most cited first, 6 of 6)

- [Towards 3D Vision with Low-Cost Single-Photon Cameras](https://mlsanity.com/paper/2403.17801/) (2024)
- [Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB](https://mlsanity.com/paper/2411.19474/) (2024)
- [Transientangelo: Few-Viewpoint Surface Reconstruction Using Single-Photon Lidar](https://mlsanity.com/paper/2408.12191/) (2024)
- [Unlocking the Performance of Proximity Sensors by Utilizing Transient Histograms](https://mlsanity.com/paper/2308.13473/) (2023)
- [Using a Distance Sensor to Detect Deviations in a Planar Surface](https://mlsanity.com/paper/2408.03838/) (2024)
- [Recovering Parametric Scenes from Very Few Time-of-Flight Pixels](https://mlsanity.com/paper/2509.16132/) (2025)

## Abstract (arXiv)

Time-resolved image sensors that capture light at pico-to-nanosecond timescales were once limited to niche applications but are now rapidly becoming mainstream in consumer devices. We propose low-cost and low-power imaging modalities that capture scene information from minimal time-resolved image sensors with as few as one pixel. The key idea is to flood illuminate large scene patches (or the entire scene) with a pulsed light source and measure the time-resolved reflected light by integrating over the entire illuminated area. The one-dimensional measured temporal waveform, called transient, encodes both distances and albedoes at all visible scene points and as such is an aggregate proxy for the scene's 3D geometry. We explore the viability and limitations of the transient waveforms by themselves for recovering scene information, and also when combined with traditional RGB cameras. We show that plane estimation can be performed from a single transient and that using only a few more it is possible to recover a depth map of the whole scene. We also show two proof-of-concept hardware prototypes that demonstrate the feasibility of our approach for compact, mobile, and budget-limited applications.