# Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry

arXiv: https://arxiv.org/abs/2409.08769 · HTML: https://mlsanity.com/paper/2409.08769/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [Delving into Deep Imbalanced Regression](https://mlsanity.com/paper/2102.09554/) (2021)
- [On-Manifold Preintegration for Real-Time Visual-Inertial Odometry](https://mlsanity.com/paper/1512.02363/) (2015)
- [VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem](https://mlsanity.com/paper/1701.08376/) (2017)
- [Selective Sensor Fusion for Neural Visual-Inertial Odometry](https://mlsanity.com/paper/1903.01534/) (2019)
- [Projective Manifold Gradient Layer for Deep Rotation Regression](https://mlsanity.com/paper/2110.11657/) (2021)
- [DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints](https://mlsanity.com/paper/1906.11435/) (2019)
- [SelfVIO: Self-Supervised Deep Monocular Visual-Inertial Odometry and Depth Estimation](https://mlsanity.com/paper/1911.09968/) (2019)
- [DM-VIO: Delayed Marginalization Visual-Inertial Odometry](https://mlsanity.com/paper/2201.04114/) (2022)
- [Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection](https://mlsanity.com/paper/2205.06187/) (2022)
- [EMA-VIO: Deep Visual-Inertial Odometry with External Memory Attention](https://mlsanity.com/paper/2209.08490/) (2022)
- [A Self-Supervised, Differentiable Kalman Filter for Uncertainty-Aware Visual-Inertial Odometry](https://mlsanity.com/paper/2203.07207/) (2022)
- [Vision-Aided Absolute Trajectory Estimation Using an Unsupervised Deep Network with Online Error Correction](https://mlsanity.com/paper/1803.05850/) (2018)
- [EqVIO: An Equivariant Filter for Visual Inertial Odometry](https://mlsanity.com/paper/2205.01980/) (2022)

## Built on by (most cited first, 3 of 3)

- [Astra: Toward General-Purpose Mobile Robots via Hierarchical Multimodal Learning](https://mlsanity.com/paper/2506.06205/) (2025)
- [DAP-Pose: Deep Temporal Alignment and Physics-aware Cross-modal Sensor Fusion for Robust Pose Estimation](https://mlsanity.com/paper/2607.23755/) (2026)
- [FFVO: A Feedforward Pose Decoder for Long-Horizon Visual Odometry](https://mlsanity.com/paper/2609.13733/) (2026)

## Abstract (arXiv)

In recent years, transformer-based architectures become the de facto standard for sequence modeling in deep learning frameworks. Inspired by the successful examples, we propose a causal visual-inertial fusion transformer (VIFT) for pose estimation in deep visual-inertial odometry. This study aims to improve pose estimation accuracy by leveraging the attention mechanisms in transformers, which better utilize historical data compared to the recurrent neural network (RNN) based methods seen in recent methods. Transformers typically require large-scale data for training. To address this issue, we utilize inductive biases for deep VIO networks. Since latent visual-inertial feature vectors encompass essential information for pose estimation, we employ transformers to refine pose estimates by updating latent vectors temporally. Our study also examines the impact of data imbalance and rotation learning methods in supervised end-to-end learning of visual inertial odometry by utilizing specialized gradients in backpropagation for the elements of SE(3) group. The proposed method is end-to-end trainable and requires only a monocular camera and IMU during inference. Experimental results demonstrate that VIFT increases the accuracy of monocular VIO networks, achieving state-of-the-art results when compared to previous methods on the KITTI dataset. The code will be made available at https://github.com/ybkurt/VIFT.