# Balancing the Budget: Feature Selection and Tracking for Multi-Camera Visual-Inertial Odometry

arXiv: https://arxiv.org/abs/2109.05975 · HTML: https://mlsanity.com/paper/2109.05975/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [Lazier Than Lazy Greedy](https://mlsanity.com/paper/1409.7938/) (2014)
- [On-Manifold Preintegration for Real-Time Visual-Inertial Odometry](https://mlsanity.com/paper/1512.02363/) (2015)
- [The Newer College Dataset: Handheld LiDAR, Inertial and Vision with Ground Truth](https://mlsanity.com/paper/2003.05691/) (2020)
- [VILENS: Visual, Inertial, Lidar, and Leg Odometry for All-Terrain Legged Robots](https://mlsanity.com/paper/2107.07243/) (2021)
- [Redesigning SLAM for Arbitrary Multi-Camera Systems](https://mlsanity.com/paper/2003.02014/) (2020)
- [Attention and Anticipation in Fast Visual-Inertial Navigation](https://mlsanity.com/paper/1610.03344/) (2016)
- [Unified Multi-Modal Landmark Tracking for Tightly Coupled Lidar-Visual-Inertial Odometry](https://mlsanity.com/paper/2011.06838/) (2020)
- [Good Feature Matching: Towards Accurate, Robust VO/VSLAM with Low Latency](https://mlsanity.com/paper/2001.00714/) (2020)
- [MIMC-VINS: A Versatile and Resilient Multi-IMU Multi-Camera Visual-Inertial Navigation System](https://mlsanity.com/paper/2006.15699/) (2020)
- [Good Feature Selection for Least Squares Pose Optimization in VO/VSLAM](https://mlsanity.com/paper/1905.07807/) (2019)

## Built on by (most cited first, 6 of 6)

- [Multi-Camera LiDAR Inertial Extension to the Newer College Dataset](https://mlsanity.com/paper/2112.08854/) (2021)
- [Language-EXtended Indoor SLAM (LEXIS): A Versatile System for Real-time Visual Scene Understanding](https://mlsanity.com/paper/2309.15065/) (2023)
- [Multi-Visual-Inertial System: Analysis, Calibration and Estimation](https://mlsanity.com/paper/2308.05303/) (2023)
- [Exosense: A Vision-Based Scene Understanding System For Exoskeletons](https://mlsanity.com/paper/2403.14320/) (2024)
- [A Pose-only Geometric Constraint for Multi-Camera Pose Adjustment](https://mlsanity.com/paper/2604.23704/) (2026)
- [Online Localisation and Colored Mesh Reconstruction Architecture for 3D Visual Feedback in Robotic Exploration Missions](https://mlsanity.com/paper/2207.10489/) (2022)

## Abstract (arXiv)

We present a multi-camera visual-inertial odometry system based on factor graph optimization which estimates motion by using all cameras simultaneously while retaining a fixed overall feature budget. We focus on motion tracking in challenging environments, such as narrow corridors, dark spaces with aggressive motions, and abrupt lighting changes. These scenarios cause traditional monocular or stereo odometry to fail. While tracking motion with extra cameras should theoretically prevent failures, it leads to additional complexity and computational burden. To overcome these challenges, we introduce two novel methods to improve multi-camera feature tracking. First, instead of tracking features separately in each camera, we track features continuously as they move from one camera to another. This increases accuracy and achieves a more compact factor graph representation. Second, we select a fixed budget of tracked features across the cameras to reduce back-end optimization time. We have found that using a smaller set of informative features can maintain the same tracking accuracy. Our proposed method was extensively tested using a hardware-synchronized device consisting of an IMU and four cameras (a front stereo pair and two lateral) in scenarios including: an underground mine, large open spaces, and building interiors with narrow stairs and corridors. Compared to stereo-only state-of-the-art visual-inertial odometry methods, our approach reduces the drift rate, relative pose error, by up to 80% in translation and 39% in rotation.