# Recovering Parametric Scenes from Very Few Time-of-Flight Pixels

arXiv: https://arxiv.org/abs/2509.16132 · HTML: https://mlsanity.com/paper/2509.16132/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022)
- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [Cameras as Relative Positional Encoding](https://mlsanity.com/paper/2507.10496/) (2025)
- [OVE6D: Object Viewpoint Encoding for Depth-based 6D Object Pose Estimation](https://mlsanity.com/paper/2203.01072/) (2022)
- [Non-line-of-sight Imaging](https://mlsanity.com/paper/2005.08026/) (2020)
- [Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor](https://mlsanity.com/paper/2308.14383/) (2023)
- [Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images](https://mlsanity.com/paper/1508.04546/) (2015)
- [A megapixel time-gated SPAD image sensor for 2D and 3D imaging applications](https://mlsanity.com/paper/1912.12910/) (2019)
- [DELTAR: Depth Estimation from a Light-weight ToF Sensor and RGB Image](https://mlsanity.com/paper/2209.13362/) (2022)
- [DART: Articulated Hand Model with Diverse Accessories and Rich Textures](https://mlsanity.com/paper/2210.07650/) (2022)
- [Photon-Flooded Single-Photon 3D Cameras](https://mlsanity.com/paper/1903.08347/) (2019)
- [Towards 3D Vision with Low-Cost Single-Photon Cameras](https://mlsanity.com/paper/2403.17801/) (2024)
- [Visual-Tactile Sensing for In-Hand Object Reconstruction](https://mlsanity.com/paper/2303.14498/) (2023)
- [Diff-DOPE: Differentiable Deep Object Pose Estimation](https://mlsanity.com/paper/2310.00463/) (2023)
- [Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB](https://mlsanity.com/paper/2411.19474/) (2024)
- [PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar](https://mlsanity.com/paper/2312.14239/) (2023)
- [Enhancing Autonomous Navigation by Imaging Hidden Objects using Single-Photon LiDAR](https://mlsanity.com/paper/2410.03555/) (2024)
- [Flying with Photons: Rendering Novel Views of Propagating Light](https://mlsanity.com/paper/2404.06493/) (2024)
- [3D Scene Inference from Transient Histograms](https://mlsanity.com/paper/2211.05094/) (2022)
- [Transientangelo: Few-Viewpoint Surface Reconstruction Using Single-Photon Lidar](https://mlsanity.com/paper/2408.12191/) (2024)
- [Unlocking the Performance of Proximity Sensors by Utilizing Transient Histograms](https://mlsanity.com/paper/2308.13473/) (2023)
- [Using a Distance Sensor to Detect Deviations in a Planar Surface](https://mlsanity.com/paper/2408.03838/) (2024)

## Built on by (most cited first, 3 of 3)

- [Spatial Calibration of Diffuse LiDARs](https://mlsanity.com/paper/2603.06531/) (2026)
- [DENALI: A Dataset Enabling Non-Line-of-Sight Spatial Reasoning with Low-Cost LiDARs](https://mlsanity.com/paper/2604.16201/) (2026)
- [Non-line-of-sight imaging with arbitrary relay surface geometries via 3D Gaussian Transient Rendering](https://mlsanity.com/paper/2606.21270/) (2026)

## Abstract (arXiv)

We aim to recover the geometry of 3D parametric scenes using very few depth measurements from low-cost, commercially available time-of-flight sensors. These sensors offer very low spatial resolution (i.e., a single pixel), but image a wide field-of-view per pixel and capture detailed time-of-flight data in the form of time-resolved photon counts. This time-of-flight data encodes rich scene information and thus enables recovery of simple scenes from sparse measurements. We investigate the feasibility of using a distributed set of few measurements (e.g., as few as 15 pixels) to recover the geometry of simple parametric scenes with a strong prior, such as estimating the 6D pose of a known object. To achieve this, we design a method that utilizes both feed-forward prediction to infer scene parameters, and differentiable rendering within an analysis-by-synthesis framework to refine the scene parameter estimate. We develop hardware prototypes and demonstrate that our method effectively recovers object pose given an untextured 3D model in both simulations and controlled real-world captures, and show promising initial results for other parametric scenes. We additionally conduct experiments to explore the limits and capabilities of our imaging solution.