# Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation

arXiv: https://arxiv.org/abs/2110.06853 · HTML: https://mlsanity.com/paper/2110.06853/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 46 references that are in the map.
- First posted on arXiv 13 Oct 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras](https://mlsanity.com/paper/1904.04998/) (2019)
- [Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos](https://mlsanity.com/paper/1811.06152/) (2018)
- [Learning Depth from Monocular Videos using Direct Methods](https://mlsanity.com/paper/1712.00175/) (2017)
- [DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency](https://mlsanity.com/paper/1809.01649/) (2018)
- [Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance](https://mlsanity.com/paper/2007.06936/) (2020)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [Semantically-Guided Representation Learning for Self-Supervised Monocular Depth](https://mlsanity.com/paper/2002.12319/) (2020)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [Kinematic 3D Object Detection in Monocular Video](https://mlsanity.com/paper/2007.09548/) (2020)
- [Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding](https://mlsanity.com/paper/1810.06125/) (2018)
- [UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning](https://mlsanity.com/paper/1709.06841/) (2017)
- [Unsupervised Monocular Depth Learning in Dynamic Scenes](https://mlsanity.com/paper/2010.16404/) (2020)
- [Self-Supervised Monocular Scene Flow Estimation](https://mlsanity.com/paper/2004.04143/) (2020)

## Built on by (most cited first, 24 of 24)

- [Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth](https://mlsanity.com/paper/2203.15174/) (2022)
- [SQLdepth: Generalizable Self-Supervised Fine-Structured Monocular Depth Estimation](https://mlsanity.com/paper/2309.00526/) (2023)
- [Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes](https://mlsanity.com/paper/2310.18887/) (2023)
- [From-Ground-To-Objects: Coarse-to-Fine Self-supervised Monocular Depth Estimation of Dynamic Objects with Ground Contact Prior](https://mlsanity.com/paper/2312.10118/) (2023)
- [SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images](https://mlsanity.com/paper/2403.10662/) (2024)
- [Manydepth2: Motion-Aware Self-Supervised Monocular Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2312.15268/) (2023)
- [Mono-ViFI: A Unified Learning Framework for Self-supervised Single- and Multi-frame Monocular Depth Estimation](https://mlsanity.com/paper/2407.14126/) (2024)
- [Embodiment: Self-Supervised Depth Estimation Based on Camera Models](https://mlsanity.com/paper/2408.01565/) (2024)
- [DS-Depth: Dynamic and Static Depth Estimation via a Fusion Cost Volume](https://mlsanity.com/paper/2308.07225/) (2023)
- [SPIdepth: Strengthened Pose Information for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2404.12501/) (2024)
- [ProDepth: Boosting Self-Supervised Multi-Frame Monocular Depth with Probabilistic Fusion](https://mlsanity.com/paper/2407.09303/) (2024)
- [Instance-aware multi-object self-supervision for monocular depth prediction](https://mlsanity.com/paper/2203.00809/) (2022)
- [MAL: Motion-Aware Loss with Temporal and Distillation Hints for Self-Supervised Depth Estimation](https://mlsanity.com/paper/2402.11507/) (2024)
- [Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2404.14908/) (2024)
- [Dyna-DepthFormer: Multi-frame Transformer for Self-Supervised Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2301.05871/) (2023)
- [Multi-Frame Self-Supervised Depth Estimation with Multi-Scale Feature Fusion in Dynamic Scenes](https://mlsanity.com/paper/2303.14628/) (2023)
- [MoDA: Leveraging Motion Priors from Videos for Advancing Unsupervised Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2309.11711/) (2023)
- [FlowDepth: Decoupling Optical Flow for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2403.19294/) (2024)
- [Pixel-Wise Prediction based Visual Odometry via Uncertainty Estimation](https://mlsanity.com/paper/2208.08892/) (2022)
- [STDepthFormer: Predicting Spatio-temporal Depth from Video with a Self-supervised Transformer Model](https://mlsanity.com/paper/2303.01196/) (2023)
- [Geometric Constraints in Deep Learning Frameworks: A Survey](https://mlsanity.com/paper/2403.12431/) (2024)
- [DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos](https://mlsanity.com/paper/2403.05895/) (2024)
- [Adaptive Discrete Disparity Volume for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2404.03190/) (2024)
- [DASP: Self-supervised Nighttime Monocular Depth Estimation with Domain Adaptation of Spatiotemporal Priors](https://mlsanity.com/paper/2512.14536/) (2025)

## Abstract (arXiv)

Estimating the motion of the camera together with the 3D structure of the scene from a monocular vision system is a complex task that often relies on the so-called scene rigidity assumption. When observing a dynamic environment, this assumption is violated which leads to an ambiguity between the ego-motion of the camera and the motion of the objects. To solve this problem, we present a self-supervised learning framework for 3D object motion field estimation from monocular videos. Our contributions are two-fold. First, we propose a two-stage projection pipeline to explicitly disentangle the camera ego-motion and the object motions with dynamics attention module, called DAM. Specifically, we design an integrated motion model that estimates the motion of the camera and object in the first and second warping stages, respectively, controlled by the attention module through a shared motion encoder. Second, we propose an object motion field estimation through contrastive sample consensus, called CSAC, taking advantage of weak semantic prior (bounding box from an object detector) and geometric constraints (each object respects the rigid body motion model). Experiments on KITTI, Cityscapes, and Waymo Open Dataset demonstrate the relevance of our approach and show that our method outperforms state-of-the-art algorithms for the tasks of self-supervised monocular depth estimation, object motion segmentation, monocular scene flow estimation, and visual odometry.