# Self-Supervised Learning of Depth and Ego-Motion from Video by Alternative Training and Geometric Constraints from 3D to 2D

arXiv: https://arxiv.org/abs/2108.01980 · HTML: https://mlsanity.com/paper/2108.01980/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 47 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [On the uncertainty of self-supervised monocular depth estimation](https://mlsanity.com/paper/2005.06209/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos](https://mlsanity.com/paper/1811.06152/) (2018)
- [Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance](https://mlsanity.com/paper/2007.06936/) (2020)
- [Feature-metric Loss for Self-supervised Learning of Depth and Egomotion](https://mlsanity.com/paper/2007.10603/) (2020)
- [Semantically-Guided Representation Learning for Self-Supervised Monocular Depth](https://mlsanity.com/paper/2002.12319/) (2020)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks](https://mlsanity.com/paper/1605.02305/) (2016)
- [Towards Better Generalization: Joint Depth-Pose Learning without PoseNet](https://mlsanity.com/paper/2004.01314/) (2020)
- [Unsupervised Monocular Depth Learning in Dynamic Scenes](https://mlsanity.com/paper/2010.16404/) (2020)
- [DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling](https://mlsanity.com/paper/2001.00987/) (2019)

## Built on by (most cited first, 1 of 1)

- [DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer](https://mlsanity.com/paper/2401.12820/) (2024)

## Abstract (arXiv)

Self-supervised learning of depth and ego-motion from unlabeled monocular video has acquired promising results and drawn extensive attention. Most existing methods jointly train the depth and pose networks by photometric consistency of adjacent frames based on the principle of structure-from-motion (SFM). However, the coupling relationship of the depth and pose networks seriously influences the learning performance, and the re-projection relations is sensitive to scale ambiguity, especially for pose learning. In this paper, we aim to improve the depth-pose learning performance without the auxiliary tasks and address the above issues by alternative training each task and incorporating the epipolar geometric constraints into the Iterative Closest Point (ICP) based point clouds match process. Distinct from jointly training the depth and pose networks, our key idea is to better utilize the mutual dependency of these two tasks by alternatively training each network with respective losses while fixing the other. We also design a log-scale 3D structural consistency loss to put more emphasis on the smaller depth values during training. To makes the optimization easier, we further incorporate the epipolar geometry into the ICP based learning process for pose learning. Extensive experiments on various benchmarks datasets indicate the superiority of our algorithm over the state-of-the-art self-supervised methods.