# Instance-wise Depth and Motion Learning from Monocular Videos

arXiv: https://arxiv.org/abs/1912.09351 · HTML: https://mlsanity.com/paper/1912.09351/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2019.

## Built on by (most cited first, 7 of 7)

- [Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation](https://mlsanity.com/paper/2110.06853/) (2021)
- [MonoGRNet: A General Framework for Monocular 3D Object Detection](https://mlsanity.com/paper/2104.08797/) (2021)
- [Multi-view Monocular Depth and Uncertainty Prediction with Deep SfM in Dynamic Environments](https://mlsanity.com/paper/2201.08633/) (2022)
- [Dyna-DepthFormer: Multi-frame Transformer for Self-Supervised Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2301.05871/) (2023)
- [Improving Neural Radiance Fields with Depth-aware Optimization for Novel View Synthesis](https://mlsanity.com/paper/2304.05218/) (2023)
- [360 in the Wild: Dataset for Depth Prediction and View Synthesis](https://mlsanity.com/paper/2406.18898/) (2024)
- [DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos](https://mlsanity.com/paper/2403.05895/) (2024)

## Abstract (arXiv)

We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular camera setup without supervision. Our technical contributions are three-fold. First, we propose a differentiable forward rigid projection module that plays a key role in our instance-wise depth and motion learning. Second, we design an instance-wise photometric and geometric consistency loss that effectively decomposes background and moving object regions. Lastly, we introduce a new auto-annotation scheme to produce video instance segmentation maps that will be utilized as input to our training pipeline. These proposed elements are validated in a detailed ablation study. Through extensive experiments conducted on the KITTI dataset, our framework is shown to outperform the state-of-the-art depth and motion estimation methods. Our code and dataset will be available at https://github.com/SeokjuLee/Insta-DM.