# Unified Multi-Modal Landmark Tracking for Tightly Coupled Lidar-Visual-Inertial Odometry

arXiv: https://arxiv.org/abs/2011.06838 · HTML: https://mlsanity.com/paper/2011.06838/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2020.

## Built on by (most cited first, 8 of 8)

- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [FAST-LIVO2: Fast, Direct LiDAR-Inertial-Visual Odometry](https://mlsanity.com/paper/2408.14035/) (2024)
- [The Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods](https://mlsanity.com/paper/2411.10546/) (2024)
- [VILENS: Visual, Inertial, Lidar, and Leg Odometry for All-Terrain Legged Robots](https://mlsanity.com/paper/2107.07243/) (2021)
- [Balancing the Budget: Feature Selection and Tracking for Multi-Camera Visual-Inertial Odometry](https://mlsanity.com/paper/2109.05975/) (2021)
- [Multi-modal curb detection and filtering](https://mlsanity.com/paper/2205.07096/) (2022)
- [Deep-MDS Framework for Recovering the 3D Shape of 2D Landmarks from a Single Image](https://mlsanity.com/paper/2210.15200/) (2022)

## Abstract (arXiv)

We present an efficient multi-sensor odometry system for mobile platforms that jointly optimizes visual, lidar, and inertial information within a single integrated factor graph. This runs in real-time at full framerate using fixed lag smoothing. To perform such tight integration, a new method to extract 3D line and planar primitives from lidar point clouds is presented. This approach overcomes the suboptimality of typical frame-to-frame tracking methods by treating the primitives as landmarks and tracking them over multiple scans. True integration of lidar features with standard visual features and IMU is made possible using a subtle passive synchronization of lidar and camera frames. The lightweight formulation of the 3D features allows for real-time execution on a single CPU. Our proposed system has been tested on a variety of platforms and scenarios, including underground exploration with a legged robot and outdoor scanning with a dynamically moving handheld device, for a total duration of 96 min and 2.4 km traveled distance. In these test sequences, using only one exteroceptive sensor leads to failure due to either underconstrained geometry (affecting lidar) or textureless areas caused by aggressive lighting changes (affecting vision). In these conditions, our factor graph naturally uses the best information available from each sensor modality without any hard switches.