# Selective Sensor Fusion for Neural Visual-Inertial Odometry

arXiv: https://arxiv.org/abs/1903.01534 · HTML: https://mlsanity.com/paper/1903.01534/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2019.

## Built on by (most cited first, 30 of 34)

- [GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer](https://mlsanity.com/paper/2406.01210/) (2024)
- [Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data](https://mlsanity.com/paper/2206.02353/) (2022)
- [Deep Learning for Inertial Positioning: A Survey](https://mlsanity.com/paper/2303.03757/) (2023)
- [Towards Scale-Aware, Robust, and Generalizable Unsupervised Monocular Depth Estimation by Integrating IMU Motion Dynamics](https://mlsanity.com/paper/2207.04680/) (2022)
- [Knowledge Perceived Multi-modal Pretraining in E-commerce](https://mlsanity.com/paper/2109.00895/) (2021)
- [Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection](https://mlsanity.com/paper/2205.06187/) (2022)
- [HydraFusion: Context-Aware Selective Sensor Fusion for Robust and Efficient Autonomous Vehicle Perception](https://mlsanity.com/paper/2201.06644/) (2022)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [EcoFusion: Energy-Aware Adaptive Sensor Fusion for Efficient Autonomous Vehicle Perception](https://mlsanity.com/paper/2202.11330/) (2022)
- [Graph-based Thermal-Inertial SLAM with Probabilistic Neural Networks](https://mlsanity.com/paper/2104.07196/) (2021)
- [From the Lab to the Wild: Affect Modeling via Privileged Information](https://mlsanity.com/paper/2305.10919/) (2023)
- [GEM: Glare or Gloom, I Can Still See You -- End-to-End Multimodal Object Detection](https://mlsanity.com/paper/2102.12319/) (2021)
- [Self-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth](https://mlsanity.com/paper/2203.01557/) (2022)
- [EMA-VIO: Deep Visual-Inertial Odometry with External Memory Attention](https://mlsanity.com/paper/2209.08490/) (2022)
- [Information-Theoretic Odometry Learning](https://mlsanity.com/paper/2203.05724/) (2022)
- [Learned Monocular Depth Priors in Visual-Inertial Initialization](https://mlsanity.com/paper/2204.09171/) (2022)
- [Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning](https://mlsanity.com/paper/2206.02856/) (2022)
- [TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation](https://mlsanity.com/paper/2304.07728/) (2023)
- [AFT-VO: Asynchronous Fusion Transformers for Multi-View Visual Odometry Estimation](https://mlsanity.com/paper/2206.12946/) (2022)
- [Parallelizing Optical Flow Estimation on an Ultra-Low Power RISC-V Cluster for Nano-UAV Navigation](https://mlsanity.com/paper/2305.13055/) (2023)
- [EffLoc: Lightweight Vision Transformer for Efficient 6-DOF Camera Relocalization](https://mlsanity.com/paper/2402.13537/) (2024)
- [Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry](https://mlsanity.com/paper/2409.08769/) (2024)
- [UL-VIO: Ultra-lightweight Visual-Inertial Odometry with Noise Robust Test-time Adaptation](https://mlsanity.com/paper/2409.13106/) (2024)
- [Multi-Camera Sensor Fusion for Visual Odometry using Deep Uncertainty Estimation](https://mlsanity.com/paper/2112.12818/) (2021)
- [Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression](https://mlsanity.com/paper/2208.00919/) (2022)
- [Data Fusion for Deep Learning on Transport Mode Detection: A Case Study](https://mlsanity.com/paper/2106.05876/) (2021)
- [Pixel-Wise Prediction based Visual Odometry via Uncertainty Estimation](https://mlsanity.com/paper/2208.08892/) (2022)
- [A survey on deep learning approaches for data integration in autonomous driving system](https://mlsanity.com/paper/2306.11740/) (2023)
- [A Consistency-Based Loss for Deep Odometry Through Uncertainty Propagation](https://mlsanity.com/paper/2107.00366/) (2021)
- [A Real-Time Fusion Framework for Long-term Visual Localization](https://mlsanity.com/paper/2210.09757/) (2022)

## Abstract (arXiv)

Deep learning approaches for Visual-Inertial Odometry (VIO) have proven successful, but they rarely focus on incorporating robust fusion strategies for dealing with imperfect input sensory data. We propose a novel end-to-end selective sensor fusion framework for monocular VIO, which fuses monocular images and inertial measurements in order to estimate the trajectory whilst improving robustness to real-life issues, such as missing and corrupted data or bad sensor synchronization. In particular, we propose two fusion modalities based on different masking strategies: deterministic soft fusion and stochastic hard fusion, and we compare with previously proposed direct fusion baselines. During testing, the network is able to selectively process the features of the available sensor modalities and produce a trajectory at scale. We present a thorough investigation on the performances on three public autonomous driving, Micro Aerial Vehicle (MAV) and hand-held VIO datasets. The results demonstrate the effectiveness of the fusion strategies, which offer better performances compared to direct fusion, particularly in presence of corrupted data. In addition, we study the interpretability of the fusion networks by visualising the masking layers in different scenarios and with varying data corruption, revealing interesting correlations between the fusion networks and imperfect sensory input data.