# Joint Depth and Normal Estimation from Real-world Time-of-flight Raw Data

arXiv: https://arxiv.org/abs/2108.03649 · HTML: https://mlsanity.com/paper/2108.03649/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 46 references that are in the map.
- First posted on arXiv 8 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [Video Depth Estimation by Fusing Flow-to-Depth Proposals](https://mlsanity.com/paper/1912.12874/) (2019)
- [Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset](https://mlsanity.com/paper/1807.10376/) (2018)
- [Deep End-to-End Alignment and Refinement for Time-of-Flight RGB-D Module](https://mlsanity.com/paper/1909.07623/) (2019)
- [DeepToF: Off-the-Shelf Real-Time Correction of Multipath Interference in Time-of-Flight Imaging](https://mlsanity.com/paper/1805.09305/) (2018)
- [Learning to Remove Multipath Distortions in Time-of-Flight Range Images for a Robotic Arm Setup](https://mlsanity.com/paper/1601.01750/) (2016)
- [SRA: Fast Removal of General Multipath for ToF Sensors](https://mlsanity.com/paper/1403.5919/) (2014)
- [Depth Sensing Beyond LiDAR Range](https://mlsanity.com/paper/2004.03048/) (2020)
- [Very Power Efficient Neural Time-of-Flight](https://mlsanity.com/paper/1812.08125/) (2018)
- [Resolving Multi-path Interference in Time-of-Flight Imaging via Modulation Frequency Diversity and Sparse Regularization](https://mlsanity.com/paper/1404.1116/) (2014)
- [High-Resolution Depth Maps Based on TOF-Stereo Fusion](https://mlsanity.com/paper/2107.14688/) (2021)

## Built on by (most cited first, 3 of 3)

- [FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras](https://mlsanity.com/paper/2210.02785/) (2022)
- [MobiFuse: A High-Precision On-device Depth Perception System with Multi-Data Fusion](https://mlsanity.com/paper/2412.13848/) (2024)
- [Towards Robust Time-of-Flight Depth Denoising with Confidence-Aware Diffusion Model](https://mlsanity.com/paper/2503.19448/) (2025)

## Abstract (arXiv)

We present a novel approach to joint depth and normal estimation for time-of-flight (ToF) sensors. Our model learns to predict the high-quality depth and normal maps jointly from ToF raw sensor data. To achieve this, we meticulously constructed the first large-scale dataset (named ToF-100) with paired raw ToF data and ground-truth high-resolution depth maps provided by an industrial depth camera. In addition, we also design a simple but effective framework for joint depth and normal estimation, applying a robust Chamfer loss via jittering to improve the performance of our model. Our experiments demonstrate that our proposed method can efficiently reconstruct high-resolution depth and normal maps and significantly outperforms state-of-the-art approaches. Our code and data will be available at urlhttps://github.com/hkustVisionRr/JointlyDepthNormalEstimation