# Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry

arXiv: https://arxiv.org/abs/1807.02570 · HTML: https://mlsanity.com/paper/1807.02570/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2018.

## Built on by (most cited first, 30 of 85)

- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences](https://mlsanity.com/paper/2507.16443/) (2025)
- [Map-free Visual Relocalization: Metric Pose Relative to a Single Image](https://mlsanity.com/paper/2210.05494/) (2022)
- [Unsupervised Scale-consistent Depth Learning from Video](https://mlsanity.com/paper/2105.11610/) (2021)
- [Behind the Scenes: Density Fields for Single View Reconstruction](https://mlsanity.com/paper/2301.07668/) (2023)
- [Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2109.12484/) (2021)
- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion](https://mlsanity.com/paper/2103.16690/) (2021)
- [TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo](https://mlsanity.com/paper/2111.07418/) (2021)
- [R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Cameras](https://mlsanity.com/paper/2308.14713/) (2023)
- [Learning Optical Flow, Depth, and Scene Flow without Real-World Labels](https://mlsanity.com/paper/2203.15089/) (2022)
- [S4C: Self-Supervised Semantic Scene Completion with Neural Fields](https://mlsanity.com/paper/2310.07522/) (2023)
- [Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2110.08192/) (2021)
- [UDepth: Fast Monocular Depth Estimation for Visually-guided Underwater Robots](https://mlsanity.com/paper/2209.12358/) (2022)
- [Gradient-based Uncertainty for Monocular Depth Estimation](https://mlsanity.com/paper/2208.02005/) (2022)
- [DF-VO: What Should Be Learnt for Visual Odometry?](https://mlsanity.com/paper/2103.00933/) (2021)
- [Kick Back & Relax: Learning to Reconstruct the World by Watching SlowTV](https://mlsanity.com/paper/2307.10713/) (2023)
- [Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter](https://mlsanity.com/paper/2208.01489/) (2022)
- [AnyCam: Learning to Recover Camera Poses and Intrinsics from Casual Videos](https://mlsanity.com/paper/2503.23282/) (2025)
- [VOLDOR: Visual Odometry from Log-logistic Dense Optical flow Residuals](https://mlsanity.com/paper/2104.06789/) (2021)
- [Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey](https://mlsanity.com/paper/2406.19675/) (2024)
- [Distill Any Depth: Distillation Creates a Stronger Monocular Depth Estimator](https://mlsanity.com/paper/2502.19204/) (2025)
- [Learning Correspondence Uncertainty via Differentiable Nonlinear Least Squares](https://mlsanity.com/paper/2305.09527/) (2023)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [The Monocular Depth Estimation Challenge](https://mlsanity.com/paper/2211.12174/) (2022)
- [Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results](https://mlsanity.com/paper/2212.06759/) (2022)
- [H-Net: Unsupervised Attention-based Stereo Depth Estimation Leveraging Epipolar Geometry](https://mlsanity.com/paper/2104.11288/) (2021)
- [Crafting Monocular Cues and Velocity Guidance for Self-Supervised Multi-Frame Depth Learning](https://mlsanity.com/paper/2208.09170/) (2022)
- [The Second Monocular Depth Estimation Challenge](https://mlsanity.com/paper/2304.07051/) (2023)
- [DRO: Deep Recurrent Optimizer for Video to Depth](https://mlsanity.com/paper/2103.13201/) (2021)

## Abstract (arXiv)

Monocular visual odometry approaches that purely rely on geometric cues are prone to scale drift and require sufficient motion parallax in successive frames for motion estimation and 3D reconstruction. In this paper, we propose to leverage deep monocular depth prediction to overcome limitations of geometry-based monocular visual odometry. To this end, we incorporate deep depth predictions into Direct Sparse Odometry (DSO) as direct virtual stereo measurements. For depth prediction, we design a novel deep network that refines predicted depth from a single image in a two-stage process. We train our network in a semi-supervised way on photoconsistency in stereo images and on consistency with accurate sparse depth reconstructions from Stereo DSO. Our deep predictions excel state-of-the-art approaches for monocular depth on the KITTI benchmark. Moreover, our Deep Virtual Stereo Odometry clearly exceeds previous monocular and deep learning based methods in accuracy. It even achieves comparable performance to the state-of-the-art stereo methods, while only relying on a single camera.