# Towards Scale Consistent Monocular Visual Odometry by Learning from the Virtual World

arXiv: https://arxiv.org/abs/2203.05712 · HTML: https://mlsanity.com/paper/2203.05712/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 11 Mar 2022.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)
- [Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things](https://mlsanity.com/paper/2011.08612/) (2020)
- [Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry](https://mlsanity.com/paper/1807.02570/) (2018)
- [Towards Better Generalization: Joint Depth-Pose Learning without PoseNet](https://mlsanity.com/paper/2004.01314/) (2020)
- [T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks](https://mlsanity.com/paper/1808.01454/) (2018)
- [Learning monocular depth estimation infusing traditional stereo knowledge](https://mlsanity.com/paper/1904.04144/) (2019)
- [Geometry-Aware Symmetric Domain Adaptation for Monocular Depth Estimation](https://mlsanity.com/paper/1904.01870/) (2019)
- [Visual Odometry Revisited: What Should Be Learnt?](https://mlsanity.com/paper/1909.09803/) (2019)
- [AdaDepth: Unsupervised Content Congruent Adaptation for Depth Estimation](https://mlsanity.com/paper/1803.01599/) (2018)
- [Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras](https://mlsanity.com/paper/1708.07878/) (2017)
- [Pseudo RGB-D for Self-Improving Monocular SLAM and Depth Prediction](https://mlsanity.com/paper/2004.10681/) (2020)
- [Self-Supervised Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2005.06136/) (2020)
- [Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry](https://mlsanity.com/paper/1904.01892/) (2019)
- [SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimation](https://mlsanity.com/paper/2006.04026/) (2020)
- [Generalizing to the Open World: Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2103.15279/) (2021)
- [Enhancing self-supervised monocular depth estimation with traditional visual odometry](https://mlsanity.com/paper/1908.03127/) (2019)
- [Deep Online Correction for Monocular Visual Odometry](https://mlsanity.com/paper/2103.10029/) (2021)

## Built on by (most cited first, 8 of 8)

- [Towards Scale-Aware, Robust, and Generalizable Unsupervised Monocular Depth Estimation by Integrating IMU Motion Dynamics](https://mlsanity.com/paper/2207.04680/) (2022)
- [Event-based Simultaneous Localization and Mapping: A Comprehensive Survey](https://mlsanity.com/paper/2304.09793/) (2023)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenes](https://mlsanity.com/paper/2207.07895/) (2022)
- [Scene-agnostic Pose Regression for Visual Localization](https://mlsanity.com/paper/2503.19543/) (2025)
- [On Robust Cross-View Consistency in Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2209.08747/) (2022)
- [Towards Accurate Ground Plane Normal Estimation from Ego-Motion](https://mlsanity.com/paper/2212.04224/) (2022)
- [CoProU-VO: Combining Projected Uncertainty for End-to-End Unsupervised Monocular Visual Odometry](https://mlsanity.com/paper/2508.00568/) (2025)

## Abstract (arXiv)

Monocular visual odometry (VO) has attracted extensive research attention by providing real-time vehicle motion from cost-effective camera images. However, state-of-the-art optimization-based monocular VO methods suffer from the scale inconsistency problem for long-term predictions. Deep learning has recently been introduced to address this issue by leveraging stereo sequences or ground-truth motions in the training dataset. However, it comes at an additional cost for data collection, and such training data may not be available in all datasets. In this work, we propose VRVO, a novel framework for retrieving the absolute scale from virtual data that can be easily obtained from modern simulation environments, whereas in the real domain no stereo or ground-truth data are required in either the training or inference phases. Specifically, we first train a scale-aware disparity network using both monocular real images and stereo virtual data. The virtual-to-real domain gap is bridged by using an adversarial training strategy to map images from both domains into a shared feature space. The resulting scale-consistent disparities are then integrated with a direct VO system by constructing a virtual stereo objective that ensures the scale consistency over long trajectories. Additionally, to address the suboptimality issue caused by the separate optimization backend and the learning process, we further propose a mutual reinforcement pipeline that allows bidirectional information flow between learning and optimization, which boosts the robustness and accuracy of each other. We demonstrate the effectiveness of our framework on the KITTI and vKITTI2 datasets.