# Deep End-to-End Alignment and Refinement for Time-of-Flight RGB-D Module

arXiv: https://arxiv.org/abs/1909.07623 · HTML: https://mlsanity.com/paper/1909.07623/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2019.

## Built on by (most cited first, 14 of 14)

- [Consistent Direct Time-of-Flight Video Depth Super-Resolution](https://mlsanity.com/paper/2211.08658/) (2022)
- [iToF2dToF: A Robust and Flexible Representation for Data-Driven Time-of-Flight Imaging](https://mlsanity.com/paper/2103.07087/) (2021)
- [Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments](https://mlsanity.com/paper/2112.03750/) (2021)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)
- [RADU: Ray-Aligned Depth Update Convolutions for ToF Data Denoising](https://mlsanity.com/paper/2111.15513/) (2021)
- [FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras](https://mlsanity.com/paper/2210.02785/) (2022)
- [Joint Depth and Normal Estimation from Real-world Time-of-flight Raw Data](https://mlsanity.com/paper/2108.03649/) (2021)
- [Robust 3D Object Detection using Probabilistic Point Clouds from Single-Photon LiDARs](https://mlsanity.com/paper/2508.00169/) (2025)
- [Towards High-Precision Depth Sensing via Monocular-Aided iToF and RGB Integration](https://mlsanity.com/paper/2508.16579/) (2025)
- [Learnable Fractional Reaction-Diffusion Dynamics for Under-Display ToF Imaging and Beyond](https://mlsanity.com/paper/2511.01704/) (2025)
- [Centimeter-Wave Free-Space Time-of-Flight Imaging](https://mlsanity.com/paper/2105.11606/) (2021)
- [MToFNet: Object Anti-Spoofing with Mobile Time-of-Flight Data](https://mlsanity.com/paper/2110.04066/) (2021)
- [Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations](https://mlsanity.com/paper/2602.18822/) (2026)
- [Reliable iToF Depth Sensing via Sensor-Intrinsic Uncertainty Modeling and State-Space Restoration](https://mlsanity.com/paper/2609.05507/) (2026)

## Abstract (arXiv)

Recently, it is increasingly popular to equip mobile RGB cameras with Time-of-Flight (ToF) sensors for active depth sensing. However, for off-the-shelf ToF sensors, one must tackle two problems in order to obtain high-quality depth with respect to the RGB camera, namely 1) online calibration and alignment; and 2) complicated error correction for ToF depth sensing. In this work, we propose a framework for jointly alignment and refinement via deep learning. First, a cross-modal optical flow between the RGB image and the ToF amplitude image is estimated for alignment. The aligned depth is then refined via an improved kernel predicting network that performs kernel normalization and applies the bias prior to the dynamic convolution. To enrich our data for end-to-end training, we have also synthesized a dataset using tools from computer graphics. Experimental results demonstrate the effectiveness of our approach, achieving state-of-the-art for ToF refinement.