# Towards Better Generalization: Joint Depth-Pose Learning without PoseNet

arXiv: https://arxiv.org/abs/2004.01314 · HTML: https://mlsanity.com/paper/2004.01314/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2020.

## Built on by (most cited first, 30 of 85)

- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo](https://mlsanity.com/paper/2109.01129/) (2021)
- [ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild](https://mlsanity.com/paper/2207.09137/) (2022)
- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [Unsupervised Scale-consistent Depth Learning from Video](https://mlsanity.com/paper/2105.11610/) (2021)
- [Channel-Wise Attention-Based Network for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2112.13047/) (2021)
- [Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Dark](https://mlsanity.com/paper/2108.03830/) (2021)
- [Fine-grained Semantics-aware Representation Enhancement for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2108.08829/) (2021)
- [SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes](https://mlsanity.com/paper/2211.03660/) (2022)
- [Toward Practical Monocular Indoor Depth Estimation](https://mlsanity.com/paper/2112.02306/) (2021)
- [Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion](https://mlsanity.com/paper/2103.16690/) (2021)
- [MonoIndoor: Towards Good Practice of Self-Supervised Monocular Depth Estimation for Indoor Environments](https://mlsanity.com/paper/2107.12429/) (2021)
- [Tame a Wild Camera: In-the-Wild Monocular Camera Calibration](https://mlsanity.com/paper/2306.10988/) (2023)
- [WorDepth: Variational Language Prior for Monocular Depth Estimation](https://mlsanity.com/paper/2404.03635/) (2024)
- [Learning Optical Flow, Depth, and Scene Flow without Real-World Labels](https://mlsanity.com/paper/2203.15089/) (2022)
- [Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2110.08192/) (2021)
- [DiffPoseNet: Direct Differentiable Camera Pose Estimation](https://mlsanity.com/paper/2203.11174/) (2022)
- [DytanVO: Joint Refinement of Visual Odometry and Motion Segmentation in Dynamic Environments](https://mlsanity.com/paper/2209.08430/) (2022)
- [DesNet: Decomposed Scale-Consistent Network for Unsupervised Depth Completion](https://mlsanity.com/paper/2211.10994/) (2022)
- [Monocular Depth Estimation through Virtual-world Supervision and Real-world SfM Self-Supervision](https://mlsanity.com/paper/2103.12209/) (2021)
- [Multi-Modal Masked Pre-Training for Monocular Panoramic Depth Completion](https://mlsanity.com/paper/2203.09855/) (2022)
- [DevNet: Self-supervised Monocular Depth Learning via Density Volume Construction](https://mlsanity.com/paper/2209.06351/) (2022)
- [RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions](https://mlsanity.com/paper/2410.02924/) (2024)
- [Towards Scale-Aware, Robust, and Generalizable Unsupervised Monocular Depth Estimation by Integrating IMU Motion Dynamics](https://mlsanity.com/paper/2207.04680/) (2022)
- [Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes](https://mlsanity.com/paper/2310.18887/) (2023)
- [RadarCam-Depth: Radar-Camera Fusion for Depth Estimation with Learned Metric Scale](https://mlsanity.com/paper/2401.04325/) (2024)
- [Learning Depth via Leveraging Semantics: Self-supervised Monocular Depth Estimation with Both Implicit and Explicit Semantic Guidance](https://mlsanity.com/paper/2102.06685/) (2021)
- [Generalizing to the Open World: Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2103.15279/) (2021)
- [XVO: Generalized Visual Odometry via Cross-Modal Self-Training](https://mlsanity.com/paper/2309.16772/) (2023)
- [Relative Pose Estimation through Affine Corrections of Monocular Depth Priors](https://mlsanity.com/paper/2501.05446/) (2025)

## Abstract (arXiv)

In this work, we tackle the essential problem of scale inconsistency for self-supervised joint depth-pose learning. Most existing methods assume that a consistent scale of depth and pose can be learned across all input samples, which makes the learning problem harder, resulting in degraded performance and limited generalization in indoor environments and long-sequence visual odometry application. To address this issue, we propose a novel system that explicitly disentangles scale from the network estimation. Instead of relying on PoseNet architecture, our method recovers relative pose by directly solving fundamental matrix from dense optical flow correspondence and makes use of a two-view triangulation module to recover an up-to-scale 3D structure. Then, we align the scale of the depth prediction with the triangulated point cloud and use the transformed depth map for depth error computation and dense reprojection check. Our whole system can be jointly trained end-to-end. Extensive experiments show that our system not only reaches state-of-the-art performance on KITTI depth and flow estimation, but also significantly improves the generalization ability of existing self-supervised depth-pose learning methods under a variety of challenging scenarios, and achieves state-of-the-art results among self-supervised learning-based methods on KITTI Odometry and NYUv2 dataset. Furthermore, we present some interesting findings on the limitation of PoseNet-based relative pose estimation methods in terms of generalization ability. Code is available at https://github.com/B1ueber2y/TrianFlow.