# VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem

arXiv: https://arxiv.org/abs/1701.08376 · HTML: https://mlsanity.com/paper/1701.08376/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2017.

## Built on by (most cited first, 30 of 51)

- [Deep Learning for Inertial Positioning: A Survey](https://mlsanity.com/paper/2303.03757/) (2023)
- [Inertial Navigation Meets Deep Learning: A Survey of Current Trends and Future Directions](https://mlsanity.com/paper/2307.00014/) (2023)
- [Towards Scale-Aware, Robust, and Generalizable Unsupervised Monocular Depth Estimation by Integrating IMU Motion Dynamics](https://mlsanity.com/paper/2207.04680/) (2022)
- [Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection](https://mlsanity.com/paper/2205.06187/) (2022)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Graph-based Thermal-Inertial SLAM with Probabilistic Neural Networks](https://mlsanity.com/paper/2104.07196/) (2021)
- [ChiNet: Deep Recurrent Convolutional Learning for Multimodal Spacecraft Pose Estimation](https://mlsanity.com/paper/2108.10282/) (2021)
- [DVI-SLAM: A Dual Visual Inertial SLAM Network](https://mlsanity.com/paper/2309.13814/) (2023)
- [HybVIO: Pushing the Limits of Real-time Visual-inertial Odometry](https://mlsanity.com/paper/2106.11857/) (2021)
- [A Review on Visual-SLAM: Advancements from Geometric Modelling to Learning-based Semantic Scene Understanding](https://mlsanity.com/paper/2209.05222/) (2022)
- [EMA-VIO: Deep Visual-Inertial Odometry with External Memory Attention](https://mlsanity.com/paper/2209.08490/) (2022)
- [Recent Advancements in Deep Learning Applications and Methods for Autonomous Navigation: A Comprehensive Review](https://mlsanity.com/paper/2302.11089/) (2023)
- [DD-VNB: A Depth-based Dual-Loop Framework for Real-time Visually Navigated Bronchoscopy](https://mlsanity.com/paper/2403.01683/) (2024)
- [VIPose: Real-time Visual-Inertial 6D Object Pose Tracking](https://mlsanity.com/paper/2107.12617/) (2021)
- [Information-Theoretic Odometry Learning](https://mlsanity.com/paper/2203.05724/) (2022)
- [Fusing Structure from Motion and Simulation-Augmented Pose Regression from Optical Flow for Challenging Indoor Environments](https://mlsanity.com/paper/2304.07250/) (2023)
- [MambaVO: Deep Visual Odometry Based on Sequential Matching Refinement and Training Smoothing](https://mlsanity.com/paper/2412.20082/) (2024)
- [GreenPCO: An Unsupervised Lightweight Point Cloud Odometry Method](https://mlsanity.com/paper/2112.04054/) (2021)
- [Learned Monocular Depth Priors in Visual-Inertial Initialization](https://mlsanity.com/paper/2204.09171/) (2022)
- [CUAHN-VIO: Content-and-Uncertainty-Aware Homography Network for Visual-Inertial Odometry](https://mlsanity.com/paper/2208.13935/) (2022)
- [Generalizable End-to-End Deep Learning Frameworks for Real-Time Attitude Estimation Using 6DoF Inertial Measurement Units](https://mlsanity.com/paper/2302.06037/) (2023)
- [TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation](https://mlsanity.com/paper/2304.07728/) (2023)
- [NeuV-SLAM: Fast Neural Multiresolution Voxel Optimization for RGBD Dense SLAM](https://mlsanity.com/paper/2402.02020/) (2024)
- [AFT-VO: Asynchronous Fusion Transformers for Multi-View Visual Odometry Estimation](https://mlsanity.com/paper/2206.12946/) (2022)
- [Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry](https://mlsanity.com/paper/2409.08769/) (2024)
- [UL-VIO: Ultra-lightweight Visual-Inertial Odometry with Noise Robust Test-time Adaptation](https://mlsanity.com/paper/2409.13106/) (2024)
- [Multi-view Monocular Depth and Uncertainty Prediction with Deep SfM in Dynamic Environments](https://mlsanity.com/paper/2201.08633/) (2022)
- [Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression](https://mlsanity.com/paper/2208.00919/) (2022)
- [Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee](https://mlsanity.com/paper/2103.02357/) (2021)
- [OdoNet: Untethered Speed Aiding for Vehicle Navigation Without Hardware Wheeled Odometer](https://mlsanity.com/paper/2109.03091/) (2021)

## Abstract (arXiv)

In this paper we present an on-manifold sequence-to-sequence learning approach to motion estimation using visual and inertial sensors. It is to the best of our knowledge the first end-to-end trainable method for visual-inertial odometry which performs fusion of the data at an intermediate feature-representation level. Our method has numerous advantages over traditional approaches. Specifically, it eliminates the need for tedious manual synchronization of the camera and IMU as well as eliminating the need for manual calibration between the IMU and camera. A further advantage is that our model naturally and elegantly incorporates domain specific information which significantly mitigates drift. We show that our approach is competitive with state-of-the-art traditional methods when accurate calibration data is available and can be trained to outperform them in the presence of calibration and synchronization errors.