# Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos

arXiv: https://arxiv.org/abs/1811.06152 · HTML: https://mlsanity.com/paper/1811.06152/

## Key facts

- Cited by 155 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2018.

## Built on by (most cited first, 30 of 155)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [Self-Supervised Monocular Depth and Ego-Motion Estimation in Endoscopy: Appearance Flow to the Rescue](https://mlsanity.com/paper/2112.08122/) (2021)
- [TAPVid-3D: A Benchmark for Tracking Any Point in 3D](https://mlsanity.com/paper/2407.05921/) (2024)
- [Unsupervised Scale-consistent Depth Learning from Video](https://mlsanity.com/paper/2105.11610/) (2021)
- [Multi-Frame Self-Supervised Depth with Transformers](https://mlsanity.com/paper/2204.07616/) (2022)
- [Channel-Wise Attention-Based Network for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2112.13047/) (2021)
- [Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency](https://mlsanity.com/paper/2102.02629/) (2021)
- [Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Dark](https://mlsanity.com/paper/2108.03830/) (2021)
- [OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Driving](https://mlsanity.com/paper/2102.07448/) (2021)
- [Consistent Depth of Moving Objects in Video](https://mlsanity.com/paper/2108.01166/) (2021)
- [Fine-grained Semantics-aware Representation Enhancement for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2108.08829/) (2021)
- [SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes](https://mlsanity.com/paper/2211.03660/) (2022)
- [Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth](https://mlsanity.com/paper/2203.15174/) (2022)
- [UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable Rendering](https://mlsanity.com/paper/2102.11870/) (2021)
- [Learning Multiple Dense Prediction Tasks from Partially Annotated Data](https://mlsanity.com/paper/2111.14893/) (2021)
- [STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimation](https://mlsanity.com/paper/2302.01334/) (2023)
- [R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes](https://mlsanity.com/paper/2108.04814/) (2021)
- [PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2112.02582/) (2021)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions](https://mlsanity.com/paper/2407.16698/) (2024)
- [Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2110.08192/) (2021)
- [MGNet: Monocular Geometric Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2206.13199/) (2022)
- [SVDistNet: Self-Supervised Near-Field Distance Estimation on Surround View Fisheye Cameras](https://mlsanity.com/paper/2104.04420/) (2021)
- [Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2108.12545/) (2021)
- [Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2304.08993/) (2023)
- [Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation](https://mlsanity.com/paper/2110.06853/) (2021)

## Abstract (arXiv)

Learning to predict scene depth from RGB inputs is a challenging task both for indoor and outdoor robot navigation. In this work we address unsupervised learning of scene depth and robot ego-motion where supervision is provided by monocular videos, as cameras are the cheapest, least restrictive and most ubiquitous sensor for robotics. Previous work in unsupervised image-to-depth learning has established strong baselines in the domain. We propose a novel approach which produces higher quality results, is able to model moving objects and is shown to transfer across data domains, e.g. from outdoors to indoor scenes. The main idea is to introduce geometric structure in the learning process, by modeling the scene and the individual objects; camera ego-motion and object motions are learned from monocular videos as input. Furthermore an online refinement method is introduced to adapt learning on the fly to unknown domains. The proposed approach outperforms all state-of-the-art approaches, including those that handle motion e.g. through learned flow. Our results are comparable in quality to the ones which used stereo as supervision and significantly improve depth prediction on scenes and datasets which contain a lot of object motion. The approach is of practical relevance, as it allows transfer across environments, by transferring models trained on data collected for robot navigation in urban scenes to indoor navigation settings. The code associated with this paper can be found at https://sites.google.com/view/struct2depth.