# Learning Interpretable BEV Based VIO without Deep Neural Networks

arXiv: https://arxiv.org/abs/2109.12292 · HTML: https://mlsanity.com/paper/2109.12292/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 25 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors](https://mlsanity.com/paper/1805.11122/) (2018)
- [Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depth](https://mlsanity.com/paper/1904.07087/) (2019)
- [Self-Supervised Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2005.06136/) (2020)
- [DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints](https://mlsanity.com/paper/1906.11435/) (2019)
- [SelfVIO: Self-Supervised Deep Monocular Visual-Inertial Odometry and Depth Estimation](https://mlsanity.com/paper/1911.09968/) (2019)
- [AVP-SLAM: Semantic Visual Mapping and Localization for Autonomous Vehicles in the Parking Lot](https://mlsanity.com/paper/2007.01813/) (2020)
- [Ground Plane based Absolute Scale Estimation for Monocular Visual Odometry](https://mlsanity.com/paper/1903.00912/) (2019)
- [RaLL: End-to-end Radar Localization on Lidar Map Using Differentiable Measurement Model](https://mlsanity.com/paper/2009.07061/) (2020)
- [Deep Phase Correlation for End-to-End Heterogeneous Sensor Measurements Matching](https://mlsanity.com/paper/2008.09474/) (2020)

## Built on by (most cited first, 2 of 2)

- [An Online Adaptation Method for Robust Depth Estimation and Visual Odometry in the Open World](https://mlsanity.com/paper/2504.11698/) (2025)
- [Brain-Inspired Visual Odometry: Balancing Speed and Interpretability through a System of Systems Approach](https://mlsanity.com/paper/2312.13162/) (2023)

## Abstract (arXiv)

Monocular visual-inertial odometry (VIO) is a critical problem in robotics and autonomous driving. Traditional methods solve this problem based on filtering or optimization. While being fully interpretable, they rely on manual interference and empirical parameter tuning. On the other hand, learning-based approaches allow for end-to-end training but require a large number of training data to learn millions of parameters. However, the non-interpretable and heavy models hinder the generalization ability. In this paper, we propose a fully differentiable, and interpretable, bird-eye-view (BEV) based VIO model for robots with local planar motion that can be trained without deep neural networks. Specifically, we first adopt Unscented Kalman Filter as a differentiable layer to predict the pitch and roll, where the covariance matrices of noise are learned to filter out the noise of the IMU raw data. Second, the refined pitch and roll are adopted to retrieve a gravity-aligned BEV image of each frame using differentiable camera projection. Finally, a differentiable pose estimator is utilized to estimate the remaining 3 DoF poses between the BEV frames: leading to a 5 DoF pose estimation. Our method allows for learning the covariance matrices end-to-end supervised by the pose estimation loss, demonstrating superior performance to empirical baselines. Experimental results on synthetic and real-world datasets demonstrate that our simple approach is competitive with state-of-the-art methods and generalizes well on unseen scenes.