# CodeVIO: Visual-Inertial Odometry with Learned Optimizable Dense Depth

arXiv: https://arxiv.org/abs/2012.10133 · HTML: https://mlsanity.com/paper/2012.10133/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2020.

## Built on by (most cited first, 23 of 23)

- [Unsupervised Depth Completion with Calibrated Backprojection Layers](https://mlsanity.com/paper/2108.10531/) (2021)
- [TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo](https://mlsanity.com/paper/2111.07418/) (2021)
- [Learning Topology from Synthetic Data for Unsupervised Depth Completion](https://mlsanity.com/paper/2106.02994/) (2021)
- [An Adaptive Framework for Learning Unsupervised Depth Completion](https://mlsanity.com/paper/2106.03010/) (2021)
- [Bayesian Deep Basis Fitting for Depth Completion with Uncertainty](https://mlsanity.com/paper/2103.15254/) (2021)
- [RadarCam-Depth: Radar-Camera Fusion for Depth Estimation with Learned Metric Scale](https://mlsanity.com/paper/2401.04325/) (2024)
- [CodeMapping: Real-Time Dense Mapping for Sparse SLAM using Compact Scene Representations](https://mlsanity.com/paper/2107.08994/) (2021)
- [Benchmarking Egocentric Visual-Inertial SLAM at City Scale](https://mlsanity.com/paper/2509.26639/) (2025)
- [NeRF-VO: Real-Time Sparse Visual Odometry with Neural Radiance Fields](https://mlsanity.com/paper/2312.13471/) (2023)
- [EMA-VIO: Deep Visual-Inertial Odometry with External Memory Attention](https://mlsanity.com/paper/2209.08490/) (2022)
- [Incremental Dense Reconstruction from Monocular Video with Guided Sparse Feature Volume Fusion](https://mlsanity.com/paper/2305.14918/) (2023)
- [Learned Monocular Depth Priors in Visual-Inertial Initialization](https://mlsanity.com/paper/2204.09171/) (2022)
- [SimpleMapping: Real-Time Visual-Inertial Dense Mapping with Deep Multi-View Stereo](https://mlsanity.com/paper/2306.08648/) (2023)
- [Sparse SPN: Depth Completion from Sparse Keypoints](https://mlsanity.com/paper/2212.00987/) (2022)
- [Depth Completion with Multiple Balanced Bases and Confidence for Dense Monocular SLAM](https://mlsanity.com/paper/2309.04145/) (2023)
- [RaCalNet: Radar Calibration Network for Sparse-Supervised Metric Depth Estimation](https://mlsanity.com/paper/2506.15560/) (2025)
- [Efficient Feature-Free Initialization for Monocular Visual-Inertial Systems Using a Feed-Forward 3D Model](https://mlsanity.com/paper/2605.17327/) (2026)
- [TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning](https://mlsanity.com/paper/2204.10993/) (2022)
- [An Improved Multi-State Constraint Kalman Filter for Visual-Inertial Odometry](https://mlsanity.com/paper/2210.08117/) (2022)
- [PRISM: Probabilistic Real-Time Inference in Spatial World Models](https://mlsanity.com/paper/2212.02988/) (2022)
- [VIMD: Monocular Visual-Inertial Motion and Depth Estimation](https://mlsanity.com/paper/2509.19713/) (2025)
- [CSMapping: Scalable Crowdsourced Semantic Mapping and Topology Inference for Autonomous Driving](https://mlsanity.com/paper/2512.03510/) (2025)
- [MFVINS: Multiple Fisheye Camera-Based Visual Inertial System](https://mlsanity.com/paper/2609.08626/) (2026)

## Abstract (arXiv)

In this work, we present a lightweight, tightly-coupled deep depth network and visual-inertial odometry (VIO) system, which can provide accurate state estimates and dense depth maps of the immediate surroundings. Leveraging the proposed lightweight Conditional Variational Autoencoder (CVAE) for depth inference and encoding, we provide the network with previously marginalized sparse features from VIO to increase the accuracy of initial depth prediction and generalization capability. The compact encoded depth maps are then updated jointly with navigation states in a sliding window estimator in order to provide the dense local scene geometry. We additionally propose a novel method to obtain the CVAE's Jacobian which is shown to be more than an order of magnitude faster than previous works, and we additionally leverage First-Estimate Jacobian (FEJ) to avoid recalculation. As opposed to previous works relying on completely dense residuals, we propose to only provide sparse measurements to update the depth code and show through careful experimentation that our choice of sparse measurements and FEJs can still significantly improve the estimated depth maps. Our full system also exhibits state-of-the-art pose estimation accuracy, and we show that it can run in real-time with single-thread execution while utilizing GPU acceleration only for the network and code Jacobian.