# AutoDepthNet: High Frame Rate Depth Map Reconstruction using Commodity Depth and RGB Cameras

arXiv: https://arxiv.org/abs/2305.14731 · HTML: https://mlsanity.com/paper/2305.14731/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 13 of 13)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation](https://mlsanity.com/paper/2204.00987/) (2022)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [FastDepth: Fast Monocular Depth Estimation on Embedded Systems](https://mlsanity.com/paper/1903.03273/) (2019)
- [Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2006.01423/) (2020)
- [ATGV-Net: Accurate Depth Super-Resolution](https://mlsanity.com/paper/1607.07988/) (2016)
- [Region Aware Video Object Segmentation with Deep Motion Modeling](https://mlsanity.com/paper/2207.10258/) (2022)
- [A Survey on RGB-D Datasets](https://mlsanity.com/paper/2201.05761/) (2022)
- [Temporal Upsampling of Depth Maps Using a Hybrid Camera](https://mlsanity.com/paper/1708.03760/) (2017)

## Built on by (most cited first, 1 of 1)

- [Empirical Studies of Large Scale Environment Scanning by Consumer Electronics](https://mlsanity.com/paper/2506.14771/) (2025)

## Abstract (arXiv)

Depth cameras have found applications in diverse fields, such as computer vision, artificial intelligence, and video gaming. However, the high latency and low frame rate of existing commodity depth cameras impose limitations on their applications. We propose a fast and accurate depth map reconstruction technique to reduce latency and increase the frame rate in depth cameras. Our approach uses only a commodity depth camera and color camera in a hybrid camera setup; our prototype is implemented using a Kinect Azure depth camera at 30 fps and a high-speed RGB iPhone 11 Pro camera captured at 240 fps. The proposed network, AutoDepthNet, is an encoder-decoder model that captures frames from the high-speed RGB camera and combines them with previous depth frames to reconstruct a stream of high frame rate depth maps. On GPU, with a 480 x 270 output resolution, our system achieves an inference time of 8 ms, enabling real-time use at up to 200 fps with parallel processing. AutoDepthNet can estimate depth values with an average RMS error of 0.076, a 44.5% improvement compared to an optical flow-based comparison method. Our method can also improve depth map quality by estimating depth values for missing and invalidated pixels. The proposed method can be easily applied to existing depth cameras and facilitates the use of depth cameras in applications that require high-speed depth estimation. We also showcase the effectiveness of the framework in upsampling different sparse datasets e.g. video object segmentation. As a demonstration of our method, we integrated our framework into existing body tracking systems and demonstrated the robustness of the proposed method in such applications.