# A Compacted Structure for Cross-domain learning on Monocular Depth and Flow Estimation

arXiv: https://arxiv.org/abs/2208.11993 · HTML: https://mlsanity.com/paper/2208.11993/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 25 Aug 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art](https://mlsanity.com/paper/1704.05519/) (2017)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency](https://mlsanity.com/paper/1809.01649/) (2018)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry](https://mlsanity.com/paper/1807.02570/) (2018)
- [Towards Better Generalization: Joint Depth-Pose Learning without PoseNet](https://mlsanity.com/paper/2004.01314/) (2020)
- [Self-Supervised Monocular Scene Flow Estimation](https://mlsanity.com/paper/2004.04143/) (2020)
- [Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids](https://mlsanity.com/paper/1604.03513/) (2016)
- [Learning Optical Flow, Depth, and Scene Flow without Real-World Labels](https://mlsanity.com/paper/2203.15089/) (2022)
- [Monocular Depth Estimation with Hierarchical Fusion of Dilated CNNs and Soft-Weighted-Sum Inference](https://mlsanity.com/paper/1708.02287/) (2017)
- [Video Depth Estimation by Fusing Flow-to-Depth Proposals](https://mlsanity.com/paper/1912.12874/) (2019)
- [Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry](https://mlsanity.com/paper/2003.00766/) (2020)
- [Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images](https://mlsanity.com/paper/2107.04644/) (2021)
- [Self-Supervised Depth Estimation in Laparoscopic Image using 3D Geometric Consistency](https://mlsanity.com/paper/2208.08407/) (2022)

## Abstract (arXiv)

Accurate motion and depth recovery is important for many robot vision tasks including autonomous driving. Most previous studies have achieved cooperative multi-task interaction via either pre-defined loss functions or cross-domain prediction. This paper presents a multi-task scheme that achieves mutual assistance by means of our Flow to Depth (F2D), Depth to Flow (D2F), and Exponential Moving Average (EMA). F2D and D2F mechanisms enable multi-scale information integration between optical flow and depth domain based on differentiable shallow nets. A dual-head mechanism is used to predict optical flow for rigid and non-rigid motion based on a divide-and-conquer manner, which significantly improves the optical flow estimation performance. Furthermore, to make the prediction more robust and stable, EMA is used for our multi-task training. Experimental results on KITTI datasets show that our multi-task scheme outperforms other multi-task schemes and provide marked improvements on the prediction results.