# Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection

arXiv: https://arxiv.org/abs/2205.06187 · HTML: https://mlsanity.com/paper/2205.06187/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 56 references that are in the map.
- First posted on arXiv 12 May 2022.

## Builds on (most cited first, 30 of 38)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [AdaFrame: Adaptive Frame Selection for Fast Video Recognition](https://mlsanity.com/paper/1811.12432/) (2018)
- [AR-Net: Adaptive Frame Resolution for Efficient Action Recognition](https://mlsanity.com/paper/2007.15796/) (2020)
- [Channel Gating Neural Networks](https://mlsanity.com/paper/1805.12549/) (2018)
- [Adaptive Focus for Efficient Video Recognition](https://mlsanity.com/paper/2105.03245/) (2021)
- [VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem](https://mlsanity.com/paper/1701.08376/) (2017)
- [Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks](https://mlsanity.com/paper/1708.06834/) (2017)
- [AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition](https://mlsanity.com/paper/2102.05775/) (2021)
- [Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling](https://mlsanity.com/paper/2007.10983/) (2020)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Selective Sensor Fusion for Neural Visual-Inertial Odometry](https://mlsanity.com/paper/1903.01534/) (2019)
- [Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry](https://mlsanity.com/paper/1904.01892/) (2019)
- [DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints](https://mlsanity.com/paper/1906.11435/) (2019)

## Built on by (most cited first, 18 of 18)

- [EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding](https://mlsanity.com/paper/2301.02217/) (2023)
- [Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomy](https://mlsanity.com/paper/2406.16087/) (2024)
- [iSLAM: Imperative SLAM](https://mlsanity.com/paper/2306.07894/) (2023)
- [Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task](https://mlsanity.com/paper/2211.05039/) (2022)
- [Diffusion-Aided Joint Source Channel Coding For High Realism Wireless Image Transmission](https://mlsanity.com/paper/2404.17736/) (2024)
- [VINGS-Mono: Visual-Inertial Gaussian Splatting Monocular SLAM in Large Scenes](https://mlsanity.com/paper/2501.08286/) (2025)
- [Recent Advancements in Deep Learning Applications and Methods for Autonomous Navigation: A Comprehensive Review](https://mlsanity.com/paper/2302.11089/) (2023)
- [EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception](https://mlsanity.com/paper/2506.21080/) (2025)
- [Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry](https://mlsanity.com/paper/2409.08769/) (2024)
- [UL-VIO: Ultra-lightweight Visual-Inertial Odometry with Noise Robust Test-time Adaptation](https://mlsanity.com/paper/2409.13106/) (2024)
- [Panopticus: Omnidirectional 3D Object Detection on Resource-constrained Edge Devices](https://mlsanity.com/paper/2410.01270/) (2024)
- [SAM-guided Pseudo Label Enhancement for Multi-modal 3D Semantic Segmentation](https://mlsanity.com/paper/2502.00960/) (2025)
- [Pseudo-LiDAR for Visual Odometry](https://mlsanity.com/paper/2209.01567/) (2022)
- [Adaptive LiDAR Scanning: Harnessing Temporal Cues for Efficient 3D Object Detection via Multi-Modal Fusion](https://mlsanity.com/paper/2508.01562/) (2025)
- [Towards Dynamic Feature Acquisition on Medical Time Series by Maximizing Conditional Mutual Information](https://mlsanity.com/paper/2407.13429/) (2024)
- [Dual-Agent Reinforcement Learning for Adaptive and Cost-Aware Visual-Inertial Odometry](https://mlsanity.com/paper/2511.21083/) (2025)
- [Uncertainty-Aware Velocity Correction for Proprioceptive Vehicle Localization using Evidential Mamba](https://mlsanity.com/paper/2607.05669/) (2026)
- [DAP-Pose: Deep Temporal Alignment and Physics-aware Cross-modal Sensor Fusion for Robust Pose Estimation](https://mlsanity.com/paper/2607.23755/) (2026)

## Abstract (arXiv)

In recent years, deep learning-based approaches for visual-inertial odometry (VIO) have shown remarkable performance outperforming traditional geometric methods. Yet, all existing methods use both the visual and inertial measurements for every pose estimation incurring potential computational redundancy. While visual data processing is much more expensive than that for the inertial measurement unit (IMU), it may not always contribute to improving the pose estimation accuracy. In this paper, we propose an adaptive deep-learning based VIO method that reduces computational redundancy by opportunistically disabling the visual modality. Specifically, we train a policy network that learns to deactivate the visual feature extractor on the fly based on the current motion state and IMU readings. A Gumbel-Softmax trick is adopted to train the policy network to make the decision process differentiable for end-to-end system training. The learned strategy is interpretable, and it shows scenario-dependent decision patterns for adaptive complexity reduction. Experiment results show that our method achieves a similar or even better performance than the full-modality baseline with up to 78.8% computational complexity reduction for KITTI dataset evaluation. The code is available at https://github.com/mingyuyng/Visual-Selective-VIO.