# Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semantics

arXiv: https://arxiv.org/abs/1906.05717 · HTML: https://mlsanity.com/paper/1906.05717/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 30 of 49)

- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2211.13202/) (2022)
- [LocalBins: Improving Depth Estimation by Learning Local Distributions](https://mlsanity.com/paper/2203.15132/) (2022)
- [Unsupervised Scale-consistent Depth Learning from Video](https://mlsanity.com/paper/2105.11610/) (2021)
- [Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency](https://mlsanity.com/paper/2102.02629/) (2021)
- [Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth](https://mlsanity.com/paper/2203.15174/) (2022)
- [Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentation](https://mlsanity.com/paper/2104.05374/) (2021)
- [Cross-View Completion Models are Zero-shot Correspondence Estimators](https://mlsanity.com/paper/2412.09072/) (2024)
- [Unsupervised Monocular Depth Estimation in Highly Complex Environments](https://mlsanity.com/paper/2107.13137/) (2021)
- [SVDistNet: Self-Supervised Near-Field Distance Estimation on Surround View Fisheye Cameras](https://mlsanity.com/paper/2104.04420/) (2021)
- [Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation](https://mlsanity.com/paper/2110.06853/) (2021)
- [SQLdepth: Generalizable Self-Supervised Fine-Structured Monocular Depth Estimation](https://mlsanity.com/paper/2309.00526/) (2023)
- [CoDEPS: Online Continual Learning for Depth Estimation and Panoptic Segmentation](https://mlsanity.com/paper/2303.10147/) (2023)
- [DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibrium](https://mlsanity.com/paper/2304.03560/) (2023)
- [An End-to-end Food Portion Estimation Framework Based on Shape Reconstruction from Monocular Image](https://mlsanity.com/paper/2308.01810/) (2023)
- [Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey](https://mlsanity.com/paper/2406.19675/) (2024)
- [Self-Supervised Monocular Depth Estimation by Direction-aware Cumulative Convolution Network](https://mlsanity.com/paper/2308.05605/) (2023)
- [SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images](https://mlsanity.com/paper/2403.10662/) (2024)
- [Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback](https://mlsanity.com/paper/2309.15762/) (2023)
- [HandsOff: Labeled Dataset Generation With No Additional Human Annotations](https://mlsanity.com/paper/2212.12645/) (2022)
- [Embodiment: Self-Supervised Depth Estimation Based on Camera Models](https://mlsanity.com/paper/2408.01565/) (2024)
- [Joint Depth Prediction and Semantic Segmentation with Multi-View SAM](https://mlsanity.com/paper/2311.00134/) (2023)
- [SPIdepth: Strengthened Pose Information for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2404.12501/) (2024)
- [LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR](https://mlsanity.com/paper/2109.03569/) (2021)
- [MonoDVPS: A Self-Supervised Monocular Depth Estimation Approach to Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2210.07577/) (2022)
- [Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes](https://mlsanity.com/paper/2403.04562/) (2024)
- [ProDepth: Boosting Self-Supervised Multi-Frame Monocular Depth with Probabilistic Fusion](https://mlsanity.com/paper/2407.09303/) (2024)
- [Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video](https://mlsanity.com/paper/2105.14520/) (2021)
- [Towards Better Data Exploitation in Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2309.05254/) (2023)
- [MAL: Motion-Aware Loss with Temporal and Distillation Hints for Self-Supervised Depth Estimation](https://mlsanity.com/paper/2402.11507/) (2024)

## Abstract (arXiv)

We present an approach which takes advantage of both structure and semantics for unsupervised monocular learning of depth and ego-motion. More specifically, we model the motion of individual objects and learn their 3D motion vector jointly with depth and ego-motion. We obtain more accurate results, especially for challenging dynamic scenes not addressed by previous approaches. This is an extended version of Casser et al. [AAAI'19]. Code and models have been open sourced at https://sites.google.com/corp/view/struct2depth.