# Depth Completion using Plane-Residual Representation

arXiv: https://arxiv.org/abs/2104.07350 · HTML: https://mlsanity.com/paper/2104.07350/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image](https://mlsanity.com/paper/1709.07492/) (2017)
- [Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera](https://mlsanity.com/paper/1807.00275/) (2018)
- [Learning Depth with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1810.02695/) (2018)
- [DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image](https://mlsanity.com/paper/1812.00488/) (2018)
- [DPSNet: End-to-end Deep Plane Sweep Stereo](https://mlsanity.com/paper/1905.00538/) (2019)
- [Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1808.00150/) (2018)
- [CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion](https://mlsanity.com/paper/1911.05377/) (2019)
- [Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints](https://mlsanity.com/paper/1910.06727/) (2019)
- [Sparse and noisy LiDAR completion with RGB guidance and uncertainty](https://mlsanity.com/paper/1902.05356/) (2019)
- [Learning Joint 2D-3D Representations for Depth Completion](https://mlsanity.com/paper/2012.12402/) (2020)
- [Fast End-to-End Trainable Guided Filter](https://mlsanity.com/paper/1803.05619/) (2018)
- [Depth Coefficients for Depth Completion](https://mlsanity.com/paper/1903.05421/) (2019)

## Built on by (most cited first, 24 of 24)

- [RigNet: Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2107.13802/) (2021)
- [TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline](https://mlsanity.com/paper/2202.08471/) (2022)
- [LRRU: Long-short Range Recurrent Updating Networks for Depth Completion](https://mlsanity.com/paper/2310.08956/) (2023)
- [Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image](https://mlsanity.com/paper/2208.13241/) (2022)
- [SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completion](https://mlsanity.com/paper/2204.13635/) (2022)
- [GraphCSPN: Geometry-Aware Depth Completion via Dynamic GCNs](https://mlsanity.com/paper/2210.10758/) (2022)
- [RGB-Depth Fusion GAN for Indoor Depth Completion](https://mlsanity.com/paper/2203.10856/) (2022)
- [Distilling Monocular Foundation Model for Fine-grained Depth Completion](https://mlsanity.com/paper/2503.16970/) (2025)
- [Depth Completion using Geometry-Aware Embedding](https://mlsanity.com/paper/2203.10912/) (2022)
- [RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2309.00655/) (2023)
- [Decomposed Guided Dynamic Filters for Efficient RGB-Guided Depth Completion](https://mlsanity.com/paper/2309.02043/) (2023)
- [Learning to Control Camera Exposure via Reinforcement Learning](https://mlsanity.com/paper/2404.01636/) (2024)
- [AGG-Net: Attention Guided Gated-convolutional Network for Depth Image Completion](https://mlsanity.com/paper/2309.01624/) (2023)
- [MIPI 2022 Challenge on RGB+ToF Depth Completion: Dataset and Report](https://mlsanity.com/paper/2209.07057/) (2022)
- [RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion](https://mlsanity.com/paper/2306.03584/) (2023)
- [Towards Domain-agnostic Depth Completion](https://mlsanity.com/paper/2207.14466/) (2022)
- [MIPI 2023 Challenge on RGB+ToF Depth Completion: Methods and Results](https://mlsanity.com/paper/2304.13916/) (2023)
- [DeCoTR: Enhancing Depth Completion with 2D and 3D Attentions](https://mlsanity.com/paper/2403.12202/) (2024)
- [Object-aware Monocular Depth Prediction with Instance Convolutions](https://mlsanity.com/paper/2112.01521/) (2021)
- [Progressive Depth Decoupling and Modulating for Flexible Depth Completion](https://mlsanity.com/paper/2405.09342/) (2024)
- [Confidence Guided Depth Completion Network](https://mlsanity.com/paper/2202.03257/) (2022)
- [Spacecraft depth completion based on the gray image and the sparse depth map](https://mlsanity.com/paper/2208.14030/) (2022)
- [Learning Pixel-wise Continuous Depth Representation via Clustering for Depth Completion](https://mlsanity.com/paper/2402.13579/) (2024)
- [ToFormer: Towards Large-scale Scenario Depth Completion for Lightweight ToF Camera](https://mlsanity.com/paper/2603.20669/) (2026)

## Abstract (arXiv)

The basic framework of depth completion is to predict a pixel-wise dense depth map using very sparse input data. In this paper, we try to solve this problem in a more effective way, by reformulating the regression-based depth estimation problem into a combination of depth plane classification and residual regression. Our proposed approach is to initially densify sparse depth information by figuring out which plane a pixel should lie among a number of discretized depth planes, and then calculate the final depth value by predicting the distance from the specified plane. This will help the network to lessen the burden of directly regressing the absolute depth information from none, and to effectively obtain more accurate depth prediction result with less computation power and inference time. To do so, we firstly introduce a novel way of interpreting depth information with the closest depth plane label p and a residual value r, as we call it, Plane-Residual (PR) representation. We also propose a depth completion network utilizing PR representation consisting of a shared encoder and two decoders, where one classifies the pixel's depth plane label, while the other one regresses the normalized distance from the classified depth plane. By interpreting depth information in PR representation and using our corresponding depth completion network, we were able to acquire improved depth completion performance with faster computation, compared to previous approaches.