# Pseudo RGB-D for Self-Improving Monocular SLAM and Depth Prediction

arXiv: https://arxiv.org/abs/2004.10681 · HTML: https://mlsanity.com/paper/2004.10681/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2020.

## Built on by (most cited first, 30 of 41)

- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)
- [Map-free Visual Relocalization: Metric Pose Relative to a Single Image](https://mlsanity.com/paper/2210.05494/) (2022)
- [Unsupervised Scale-consistent Depth Learning from Video](https://mlsanity.com/paper/2105.11610/) (2021)
- [MonoIndoor: Towards Good Practice of Self-Supervised Monocular Depth Estimation for Indoor Environments](https://mlsanity.com/paper/2107.12429/) (2021)
- [PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2203.12082/) (2022)
- [SAGE: SLAM with Appearance and Geometry Prior for Endoscopy](https://mlsanity.com/paper/2202.09487/) (2022)
- [DesNet: Decomposed Scale-Consistent Network for Unsupervised Depth Completion](https://mlsanity.com/paper/2211.10994/) (2022)
- [Beyond Image to Depth: Improving Depth Prediction using Echoes](https://mlsanity.com/paper/2103.08468/) (2021)
- [Improving Monocular Visual Odometry Using Learned Depth](https://mlsanity.com/paper/2204.01268/) (2022)
- [Adversarial Training of Self-supervised Monocular Depth Estimation against Physical-World Attacks](https://mlsanity.com/paper/2301.13487/) (2023)
- [LightDepth: Single-View Depth Self-Supervision from Illumination Decline](https://mlsanity.com/paper/2308.10525/) (2023)
- [MonoIndoor++:Towards Better Practice of Self-Supervised Monocular Depth Estimation for Indoor Environments](https://mlsanity.com/paper/2207.08951/) (2022)
- [Photometric single-view dense 3D reconstruction in endoscopy](https://mlsanity.com/paper/2204.09083/) (2022)
- [SfM-TTR: Using Structure from Motion for Test-Time Refinement of Single-View Depth Networks](https://mlsanity.com/paper/2211.13551/) (2022)
- [Fusing the Old with the New: Learning Relative Camera Pose with Geometry-Guided Uncertainty](https://mlsanity.com/paper/2104.08278/) (2021)
- [Towards Scale Consistent Monocular Visual Odometry by Learning from the Virtual World](https://mlsanity.com/paper/2203.05712/) (2022)
- [GeoRefine: Self-Supervised Online Depth Refinement for Accurate Dense Mapping](https://mlsanity.com/paper/2205.01656/) (2022)
- [Self-supervised Adversarial Training of Monocular Depth Estimation against Physical-World Attacks](https://mlsanity.com/paper/2406.05857/) (2024)
- [MAL: Motion-Aware Loss with Temporal and Distillation Hints for Self-Supervised Depth Estimation](https://mlsanity.com/paper/2402.11507/) (2024)
- [Bayesian Deep Neural Networks for Supervised Learning of Single-View Depth](https://mlsanity.com/paper/2104.14202/) (2021)
- [Self-Supervised Depth Completion for Active Stereo](https://mlsanity.com/paper/2110.03234/) (2021)
- [Monocular Depth Distribution Alignment with Low Computation](https://mlsanity.com/paper/2203.04538/) (2022)
- [FisheyeDistill: Self-Supervised Monocular Depth Estimation with Ordinal Distillation for Fisheye Cameras](https://mlsanity.com/paper/2205.02930/) (2022)
- [Variational Structured Attention Networks for Deep Visual Representation Learning](https://mlsanity.com/paper/2103.03510/) (2021)
- [RAUM-VO: Rotational Adjusted Unsupervised Monocular Visual Odometry](https://mlsanity.com/paper/2203.07162/) (2022)
- [Semantics-Depth-Symbiosis: Deeply Coupled Semi-Supervised Learning of Semantics and Depth](https://mlsanity.com/paper/2206.10562/) (2022)
- [Pseudo-LiDAR for Visual Odometry](https://mlsanity.com/paper/2209.01567/) (2022)
- [HDPV-SLAM: Hybrid Depth-augmented Panoramic Visual SLAM for Mobile Mapping System with Tilted LiDAR and Panoramic Visual Camera](https://mlsanity.com/paper/2301.11823/) (2023)
- [BaseBoostDepth: Exploiting Larger Baselines For Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2407.20437/) (2024)
- [Towards Continual, Online, Self-Supervised Depth](https://mlsanity.com/paper/2103.00369/) (2021)

## Abstract (arXiv)

Classical monocular Simultaneous Localization And Mapping (SLAM) and the recently emerging convolutional neural networks (CNNs) for monocular depth prediction represent two largely disjoint approaches towards building a 3D map of the surrounding environment. In this paper, we demonstrate that the coupling of these two by leveraging the strengths of each mitigates the other's shortcomings. Specifically, we propose a joint narrow and wide baseline based self-improving framework, where on the one hand the CNN-predicted depth is leveraged to perform pseudo RGB-D feature-based SLAM, leading to better accuracy and robustness than the monocular RGB SLAM baseline. On the other hand, the bundle-adjusted 3D scene structures and camera poses from the more principled geometric SLAM are injected back into the depth network through novel wide baseline losses proposed for improving the depth prediction network, which then continues to contribute towards better pose and 3D structure estimation in the next iteration. We emphasize that our framework only requires unlabeled monocular videos in both training and inference stages, and yet is able to outperform state-of-the-art self-supervised monocular and stereo depth prediction networks (e.g, Monodepth2) and feature-based monocular SLAM system (i.e, ORB-SLAM). Extensive experiments on KITTI and TUM RGB-D datasets verify the superiority of our self-improving geometry-CNN framework.