# Accurate Ground-Truth Depth Image Generation via Overfit Training of Point Cloud Registration using Local Frame Sets

arXiv: https://arxiv.org/abs/2207.07016 · HTML: https://mlsanity.com/paper/2207.07016/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 57 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 30 of 32)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [RPM-Net: Robust Point Matching using Learned Features](https://mlsanity.com/paper/2003.13479/) (2020)
- [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018)
- [BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration](https://mlsanity.com/paper/1604.01093/) (2016)
- [Deep Depth Completion of a Single RGB-D Image](https://mlsanity.com/paper/1803.09326/) (2018)
- [Feature-metric Registration: A Fast Semi-supervised Approach for Robust Point Cloud Registration without Correspondences](https://mlsanity.com/paper/2005.01014/) (2020)
- [3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration](https://mlsanity.com/paper/1807.09413/) (2018)
- [To learn image super-resolution, use a GAN to learn how to do image degradation first](https://mlsanity.com/paper/1807.11458/) (2018)
- [Learning multiview 3D point cloud registration](https://mlsanity.com/paper/2001.05119/) (2020)
- [USIP: Unsupervised Stable Interest Point Detection from 3D Point Clouds](https://mlsanity.com/paper/1904.00229/) (2019)
- [Multimodal Deep Learning for Robust RGB-D Object Recognition](https://mlsanity.com/paper/1507.06821/) (2015)
- [End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds](https://mlsanity.com/paper/2003.05855/) (2020)
- [UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable Rendering](https://mlsanity.com/paper/2102.11870/) (2021)
- [3D Local Features for Direct Pairwise Registration](https://mlsanity.com/paper/1904.04281/) (2019)
- [R-PointHop: A Green, Accurate, and Unsupervised Point Cloud Registration Method](https://mlsanity.com/paper/2103.08129/) (2021)
- [Sampling Network Guided Cross-Entropy Method for Unsupervised Point Cloud Registration](https://mlsanity.com/paper/2109.06619/) (2021)
- [DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition](https://mlsanity.com/paper/1702.08558/) (2017)
- [Self-Supervised Deep Depth Denoising](https://mlsanity.com/paper/1909.01193/) (2019)
- [Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertainty](https://mlsanity.com/paper/1608.02239/) (2016)
- [RGBD Datasets: Past, Present and Future](https://mlsanity.com/paper/1604.00999/) (2016)

## Abstract (arXiv)

Accurate three-dimensional perception is a fundamental task in several computer vision applications. Recently, commercial RGB-depth (RGB-D) cameras have been widely adopted as single-view depth-sensing devices owing to their efficient depth-sensing abilities. However, the depth quality of most RGB-D sensors remains insufficient owing to the inherent noise from a single-view environment. Recently, several studies have focused on the single-view depth enhancement of RGB-D cameras. Recent research has proposed deep-learning-based approaches that typically train networks using high-quality supervised depth datasets, which indicates that the quality of the ground-truth (GT) depth dataset is a top-most important factor for accurate system; however, such high-quality GT datasets are difficult to obtain. In this study, we developed a novel method for high-quality GT depth generation based on an RGB-D stream dataset. First, we defined consecutive depth frames in a local spatial region as a local frame set. Then, the depth frames were aligned to a certain frame in the local frame set using an unsupervised point cloud registration scheme. The registration parameters were trained based on an overfit-training scheme, which was primarily used to construct a single GT depth image for each frame set. The final GT depth dataset was constructed using several local frame sets, and each local frame set was trained independently. The primary advantage of this study is that a high-quality GT depth dataset can be constructed under various scanning environments using only the RGB-D stream dataset. Moreover, our proposed method can be used as a new benchmark GT dataset for accurate performance evaluations. We evaluated our GT dataset on previously benchmarked GT depth datasets and demonstrated that our method is superior to state-of-the-art depth enhancement frameworks.