# Robust Localization with Visual-Inertial Odometry Constraints for Markerless Mobile AR

arXiv: https://arxiv.org/abs/2308.05394 · HTML: https://mlsanity.com/paper/2308.05394/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [Modelling Uncertainty in Deep Learning for Camera Relocalization](https://mlsanity.com/paper/1509.05909/) (2015)
- [LENS: Localization enhanced by NeRF synthesis](https://mlsanity.com/paper/2110.06558/) (2021)
- [Learning Multi-Scene Absolute Pose Regression with Transformers](https://mlsanity.com/paper/2103.11468/) (2021)
- [LaMAR: Benchmarking Localization and Mapping for Augmented Reality](https://mlsanity.com/paper/2210.10770/) (2022)
- [DFNet: Enhance Absolute Pose Regression with Direct Feature Matching](https://mlsanity.com/paper/2204.00559/) (2022)
- [Image-based Localization using Hourglass Networks](https://mlsanity.com/paper/1703.07971/) (2017)
- [On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation](https://mlsanity.com/paper/2109.00524/) (2021)
- [Learning to Navigate the Energy Landscape](https://mlsanity.com/paper/1603.05772/) (2016)
- [VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry](https://mlsanity.com/paper/1804.08366/) (2018)
- [Direct-PoseNet: Absolute Pose Regression with Photometric Consistency](https://mlsanity.com/paper/2104.04073/) (2021)
- [Deep Auxiliary Learning for Visual Localization and Odometry](https://mlsanity.com/paper/1803.03642/) (2018)
- [Deep Bingham Networks: Dealing with Uncertainty and Ambiguity in Pose Estimation](https://mlsanity.com/paper/2012.11002/) (2020)
- [Project AutoVision: Localization and 3D Scene Perception for an Autonomous Vehicle with a Multi-Camera System](https://mlsanity.com/paper/1809.05477/) (2018)
- [CoordiNet: uncertainty-aware pose regressor for reliable vehicle localization](https://mlsanity.com/paper/2103.10796/) (2021)
- [ADVIO: An authentic dataset for visual-inertial odometry](https://mlsanity.com/paper/1807.09828/) (2018)
- [6D Camera Relocalization in Ambiguous Scenes via Continuous Multimodal Inference](https://mlsanity.com/paper/2004.04807/) (2020)
- [A Probabilistic Framework for Visual Localization in Ambiguous Scenes](https://mlsanity.com/paper/2301.02086/) (2023)
- [Improving Feature-based Visual Localization by Geometry-Aided Matching](https://mlsanity.com/paper/2211.08712/) (2022)

## Abstract (arXiv)

Visual Inertial Odometry (VIO) is an essential component of modern Augmented Reality (AR) applications. However, VIO only tracks the relative pose of the device, leading to drift over time. Absolute pose estimation methods infer the device's absolute pose, but their accuracy depends on the input quality. This paper introduces VIO-APR, a new framework for markerless mobile AR that combines an absolute pose regressor (APR) with a local VIO tracking system. VIO-APR uses VIO to assess the reliability of the APR and the APR to identify and compensate for VIO drift. This feedback loop results in more accurate positioning and more stable AR experiences. To evaluate VIO-APR, we created a dataset that combines camera images with ARKit's VIO system output for six indoor and outdoor scenes of various scales. Over this dataset, VIO-APR improves the median accuracy of popular APR by up to 36% in position and 29% in orientation, increases the percentage of frames in the high (0.25 m, 2circ) accuracy level by up to 112% and reduces the percentage of frames predicted below the low (5 m, 10^circ) accuracy greatly. We implement VIO-APR into a mobile AR application using Unity to demonstrate its capabilities. VIO-APR results in noticeably more accurate localization and a more stable overall experience.