# DeepToF: Off-the-Shelf Real-Time Correction of Multipath Interference in Time-of-Flight Imaging

arXiv: https://arxiv.org/abs/1805.09305 · HTML: https://mlsanity.com/paper/1805.09305/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2018.

## Built on by (most cited first, 13 of 13)

- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [iToF2dToF: A Robust and Flexible Representation for Data-Driven Time-of-Flight Imaging](https://mlsanity.com/paper/2103.07087/) (2021)
- [Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments](https://mlsanity.com/paper/2112.03750/) (2021)
- [Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging](https://mlsanity.com/paper/2103.16693/) (2021)
- [High-speed object detection with a single-photon time-of-flight image sensor](https://mlsanity.com/paper/2107.13407/) (2021)
- [Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging](https://mlsanity.com/paper/2204.09871/) (2022)
- [RADU: Ray-Aligned Depth Update Convolutions for ToF Data Denoising](https://mlsanity.com/paper/2111.15513/) (2021)
- [Joint Depth and Normal Estimation from Real-world Time-of-flight Raw Data](https://mlsanity.com/paper/2108.03649/) (2021)
- [Self-Annotated 3D Geometric Learning for Smeared Points Removal](https://mlsanity.com/paper/2311.09029/) (2023)
- [Swept-Angle Synthetic Wavelength Interferometry](https://mlsanity.com/paper/2205.10655/) (2022)
- [Lightweight Deep Learning Architecture for MPI Correction and Transient Reconstruction](https://mlsanity.com/paper/2111.14396/) (2021)
- [Weakly-Supervised Optical Flow Estimation for Time-of-Flight](https://mlsanity.com/paper/2210.05298/) (2022)
- [MPI Planar Correction of Pulse Based ToF Cameras](https://mlsanity.com/paper/2312.12064/) (2023)

## Abstract (arXiv)

Time-of-flight (ToF) imaging has become a widespread technique for depth estimation, allowing affordable off-the-shelf cameras to provide depth maps in real time. However, multipath interference (MPI) resulting from indirect illumination significantly degrades the captured depth. Most previous works have tried to solve this problem by means of complex hardware modifications or costly computations. In this work we avoid these approaches, and propose a new technique that corrects errors in depth caused by MPI that requires no camera modifications, and corrects depth in just 10 milliseconds per frame. By observing that most MPI information can be expressed as a function of the captured depth, we pose MPI removal as a convolutional approach, and model it using a convolutional neural network. In particular, given that the input and output data present similar structure, we base our network in an autoencoder, which we train in two stages: first, we use the encoder (convolution filters) to learn a suitable basis to represent corrupted range images; then, we train the decoder (deconvolution filters) to correct depth from the learned basis from synthetically generated scenes. This approach allows us to tackle the lack of reference data, by using a large-scale captured training set with corrupted depth to train the encoder, and a smaller synthetic training set with ground truth depth to train the corrector stage of the network, which we generate by using a physically-based, time-resolved rendering. We demonstrate and validate our method on both synthetic and real complex scenarios, using an off-the-shelf ToF camera, and with only the captured incorrect depth as input.