# Generalizing to the Open World: Deep Visual Odometry with Online Adaptation

arXiv: https://arxiv.org/abs/2103.15279 · HTML: https://mlsanity.com/paper/2103.15279/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos](https://mlsanity.com/paper/1811.06152/) (2018)
- [DeepTAM: Deep Tracking and Mapping](https://mlsanity.com/paper/1808.01900/) (2018)
- [Towards Better Generalization: Joint Depth-Pose Learning without PoseNet](https://mlsanity.com/paper/2004.01314/) (2020)
- [Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation](https://mlsanity.com/paper/1704.02157/) (2017)
- [Visual Odometry Revisited: What Should Be Learnt?](https://mlsanity.com/paper/1909.09803/) (2019)
- [Streaming Variational Bayes](https://mlsanity.com/paper/1307.6769/) (2013)
- [P2Net: Patch-match and Plane-regularization for Unsupervised Indoor Depth Estimation](https://mlsanity.com/paper/2007.07696/) (2020)
- [Self-Supervised Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2005.06136/) (2020)
- [Moving Indoor: Unsupervised Video Depth Learning in Challenging Environments](https://mlsanity.com/paper/1910.08898/) (2019)
- [Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry](https://mlsanity.com/paper/1904.01892/) (2019)
- [Open-World Stereo Video Matching with Deep RNN](https://mlsanity.com/paper/1808.03959/) (2018)
- [Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry](https://mlsanity.com/paper/1908.08704/) (2019)

## Built on by (most cited first, 19 of 19)

- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimation](https://mlsanity.com/paper/2302.01334/) (2023)
- [Learning Optical Flow, Depth, and Scene Flow without Real-World Labels](https://mlsanity.com/paper/2203.15089/) (2022)
- [XVO: Generalized Visual Odometry via Cross-Modal Self-Training](https://mlsanity.com/paper/2309.16772/) (2023)
- [Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models](https://mlsanity.com/paper/2603.22212/) (2026)
- [SC-wLS: Towards Interpretable Feed-forward Camera Re-localization](https://mlsanity.com/paper/2210.12748/) (2022)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [Towards Scale Consistent Monocular Visual Odometry by Learning from the Virtual World](https://mlsanity.com/paper/2203.05712/) (2022)
- [GeoRefine: Self-Supervised Online Depth Refinement for Accurate Dense Mapping](https://mlsanity.com/paper/2205.01656/) (2022)
- [CCL: Continual Contrastive Learning for LiDAR Place Recognition](https://mlsanity.com/paper/2303.13952/) (2023)
- [Scene-agnostic Pose Regression for Visual Localization](https://mlsanity.com/paper/2503.19543/) (2025)
- [DeepMLE: A Robust Deep Maximum Likelihood Estimator for Two-view Structure from Motion](https://mlsanity.com/paper/2210.05517/) (2022)
- [RAUM-VO: Rotational Adjusted Unsupervised Monocular Visual Odometry](https://mlsanity.com/paper/2203.07162/) (2022)
- [An Online Adaptation Method for Robust Depth Estimation and Visual Odometry in the Open World](https://mlsanity.com/paper/2504.11698/) (2025)
- [FusionDepth: Complement Self-Supervised Monocular Depth Estimation with Cost Volume](https://mlsanity.com/paper/2305.06036/) (2023)
- [Online Adaptive Disparity Estimation for Dynamic Scenes in Structured Light Systems](https://mlsanity.com/paper/2310.08934/) (2023)
- [Dense-depth map guided deep Lidar-Visual Odometry with Sparse Point Clouds and Images](https://mlsanity.com/paper/2507.15496/) (2025)

## Abstract (arXiv)

Despite learning-based visual odometry (VO) has shown impressive results in recent years, the pretrained networks may easily collapse in unseen environments. The large domain gap between training and testing data makes them difficult to generalize to new scenes. In this paper, we propose an online adaptation framework for deep VO with the assistance of scene-agnostic geometric computations and Bayesian inference. In contrast to learning-based pose estimation, our method solves pose from optical flow and depth while the single-view depth estimation is continuously improved with new observations by online learned uncertainties. Meanwhile, an online learned photometric uncertainty is used for further depth and pose optimization by a differentiable Gauss-Newton layer. Our method enables fast adaptation of deep VO networks to unseen environments in a self-supervised manner. Extensive experiments including Cityscapes to KITTI and outdoor KITTI to indoor TUM demonstrate that our method achieves state-of-the-art generalization ability among self-supervised VO methods.