# Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry

arXiv: https://arxiv.org/abs/2003.00766 · HTML: https://mlsanity.com/paper/2003.00766/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2020.

## Built on by (most cited first, 21 of 21)

- [PanoFlow: Learning 360deg Optical Flow for Surrounding Temporal Understanding](https://mlsanity.com/paper/2202.13388/) (2022)
- [Efficient 3D Deep LiDAR Odometry](https://mlsanity.com/paper/2111.02135/) (2021)
- [Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey](https://mlsanity.com/paper/2406.19675/) (2024)
- [Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network](https://mlsanity.com/paper/2412.15678/) (2024)
- [SC-NeRF: Self-Correcting Neural Radiance Field with Sparse Views](https://mlsanity.com/paper/2309.05028/) (2023)
- [3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video](https://mlsanity.com/paper/2112.03045/) (2021)
- [Unsupervised Learning of 3D Scene Flow from Monocular Camera](https://mlsanity.com/paper/2206.03673/) (2022)
- [Unsupervised Monocular Depth Perception: Focusing on Moving Objects](https://mlsanity.com/paper/2108.13062/) (2021)
- [Outdoor Monocular Depth Estimation: A Research Review](https://mlsanity.com/paper/2205.01399/) (2022)
- [NccFlow: Unsupervised Learning of Optical Flow With Non-occlusion from Geometry](https://mlsanity.com/paper/2107.03610/) (2021)
- [JointSplat: Probabilistic Joint Flow-Depth Optimization for Sparse-View Gaussian Splatting](https://mlsanity.com/paper/2506.03872/) (2025)
- [Pseudo-LiDAR for Visual Odometry](https://mlsanity.com/paper/2209.01567/) (2022)
- [CbwLoss: Constrained Bidirectional Weighted Loss for Self-supervised Learning of Depth and Pose](https://mlsanity.com/paper/2212.05845/) (2022)
- [ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation](https://mlsanity.com/paper/2309.14744/) (2023)
- [HQDec: Self-Supervised Monocular Depth Estimation Based on a High-Quality Decoder](https://mlsanity.com/paper/2305.18706/) (2023)
- [UFD-PRiME: Unsupervised Joint Learning of Optical Flow and Stereo Depth through Pixel-Level Rigid Motion Estimation](https://mlsanity.com/paper/2310.04712/) (2023)
- [Physically-Based Photometric Bundle Adjustment in Non-Lambertian Environments](https://mlsanity.com/paper/2409.11854/) (2024)
- [A Compacted Structure for Cross-domain learning on Monocular Depth and Flow Estimation](https://mlsanity.com/paper/2208.11993/) (2022)
- [Timeliness-Fidelity Tradeoff in 3D Scene Representations](https://mlsanity.com/paper/2407.16575/) (2024)
- [Task-Oriented Communications for 3D Scene Representation: Balancing Timeliness and Fidelity](https://mlsanity.com/paper/2509.17282/) (2025)
- [DAPM: UAV Monocular Depth Estimation from Any Height, Pitch, Roll and FOV](https://mlsanity.com/paper/2607.21438/) (2026)

## Abstract (arXiv)

In autonomous driving, monocular sequences contain lots of information. Monocular depth estimation, camera ego-motion estimation and optical flow estimation in consecutive frames are high-profile concerns recently. By analyzing tasks above, pixels in the middle frame are modeled into three parts: the rigid region, the non-rigid region, and the occluded region. In joint unsupervised training of depth and pose, we can segment the occluded region explicitly. The occlusion information is used in unsupervised learning of depth, pose and optical flow, as the image reconstructed by depth-pose and optical flow will be invalid in occluded regions. A less-than-mean mask is designed to further exclude the mismatched pixels interfered with by motion or illumination change in the training of depth and pose networks. This method is also used to exclude some trivial mismatched pixels in the training of the optical flow network. Maximum normalization is proposed for depth smoothness term to restrain depth degradation in textureless regions. In the occluded region, as depth and camera motion can provide more reliable motion estimation, they can be used to instruct unsupervised learning of optical flow. Our experiments in KITTI dataset demonstrate that the model based on three regions, full and explicit segmentation of the occlusion region, the rigid region, and the non-rigid region with corresponding unsupervised losses can improve performance on three tasks significantly. The source code is available at: https://github.com/guangmingw/DOPlearning.