# VILENS: Visual, Inertial, Lidar, and Leg Odometry for All-Terrain Legged Robots

arXiv: https://arxiv.org/abs/2107.07243 · HTML: https://mlsanity.com/paper/2107.07243/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 71 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2007.00258/) (2020)
- [On-Manifold Preintegration for Real-Time Visual-Inertial Odometry](https://mlsanity.com/paper/1512.02363/) (2015)
- [The Newer College Dataset: Handheld LiDAR, Inertial and Vision with Ground Truth](https://mlsanity.com/paper/2003.05691/) (2020)
- [RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control](https://mlsanity.com/paper/2012.03094/) (2020)
- [An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Research](https://mlsanity.com/paper/1910.00093/) (2019)
- [Contact-Aided Invariant Extended Kalman Filtering for Robot State Estimation](https://mlsanity.com/paper/1904.09251/) (2019)
- [LOCUS: A Multi-Sensor Lidar-Centric Solution for High-Precision Odometry and 3D Mapping in Real-Time](https://mlsanity.com/paper/2012.14447/) (2020)
- [Legged Robot State Estimation using Invariant Kalman Filtering and Learned Contact Events](https://mlsanity.com/paper/2106.15713/) (2021)
- [Online LiDAR-SLAM for Legged Robots with Robust Registration and Deep-Learned Loop Closure](https://mlsanity.com/paper/2001.10249/) (2020)
- [Receding-Horizon Perceptive Trajectory Optimization for Dynamic Legged Locomotion with Learned Initialization](https://mlsanity.com/paper/2104.09078/) (2021)
- [Unified Multi-Modal Landmark Tracking for Tightly Coupled Lidar-Visual-Inertial Odometry](https://mlsanity.com/paper/2011.06838/) (2020)
- [Hybrid Contact Preintegration for Visual-Inertial-Contact State Estimation Using Factor Graphs](https://mlsanity.com/paper/1803.07531/) (2018)
- [Robust Legged Robot State Estimation Using Factor Graph Optimization](https://mlsanity.com/paper/1904.03048/) (2019)
- [Legged Robot State-Estimation Through Combined Forward Kinematic and Preintegrated Contact Factors](https://mlsanity.com/paper/1712.05873/) (2017)
- [A Factor-Graph Approach for Optimization Problems with Dynamics Constraints](https://mlsanity.com/paper/2011.06194/) (2020)
- [On State Estimation for Legged Locomotion over Soft Terrain](https://mlsanity.com/paper/2101.02279/) (2021)
- [Preintegrated Velocity Bias Estimation to Overcome Contact Nonlinearities in Legged Robot Odometry](https://mlsanity.com/paper/1910.09875/) (2019)

## Built on by (most cited first, 25 of 25)

- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Visual Whole-Body Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2403.16967/) (2024)
- [The Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods](https://mlsanity.com/paper/2411.10546/) (2024)
- [The Hilti SLAM Challenge Dataset](https://mlsanity.com/paper/2109.11316/) (2021)
- [SiLVR: Scalable Lidar-Visual Reconstruction with Neural Radiance Fields for Robotic Inspection](https://mlsanity.com/paper/2403.06877/) (2024)
- [SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience](https://mlsanity.com/paper/2409.13678/) (2024)
- [QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robots](https://mlsanity.com/paper/2508.02512/) (2025)
- [Balancing the Budget: Feature Selection and Tracking for Multi-Camera Visual-Inertial Odometry](https://mlsanity.com/paper/2109.05975/) (2021)
- [Legged Robot State Estimation With Invariant Extended Kalman Filter Using Neural Measurement Network](https://mlsanity.com/paper/2402.00366/) (2024)
- [R3LIVE++: A Robust, Real-time, Radiance reconstruction package with a tightly-coupled LiDAR-Inertial-Visual state Estimator](https://mlsanity.com/paper/2209.03666/) (2022)
- [SiLVR: Scalable Lidar-Visual Radiance Field Reconstruction with Uncertainty Quantification](https://mlsanity.com/paper/2502.02657/) (2025)
- [3D Freehand Ultrasound using Visual Inertial and Deep Inertial Odometry for Measuring Patellar Tracking](https://mlsanity.com/paper/2404.15847/) (2024)
- [CrossTracker: Robust Multi-modal 3D Multi-Object Tracking via Cross Correction](https://mlsanity.com/paper/2411.18850/) (2024)
- [Seg2Box: 3D Object Detection by Point-Wise Semantics Supervision](https://mlsanity.com/paper/2503.16811/) (2025)
- [Robust Localization, Mapping, and Navigation for Quadruped Robots](https://mlsanity.com/paper/2505.02272/) (2025)
- [Kinematically-Decoupled Impedance Control for Fast Object Visual Servoing and Grasping on Quadruped Manipulators](https://mlsanity.com/paper/2307.04918/) (2023)
- [Holistic Fusion: Task- and Setup-Agnostic Robot Localization and State Estimation with Factor Graphs](https://mlsanity.com/paper/2504.06479/) (2025)
- [Tightly-Coupled LiDAR-IMU-Leg Odometry with Online Learned Leg Kinematics Incorporating Foot Tactile Information](https://mlsanity.com/paper/2506.09548/) (2025)
- [TreeLoc: 6-DoF LiDAR Global Localization in Forests via Inter-Tree Geometric Matching](https://mlsanity.com/paper/2602.01501/) (2026)
- [KING: Embodiment-Aware Kinematic Graph Neural Network for Unified Motion Representation of Legged and Wheeled Robots](https://mlsanity.com/paper/2608.01015/) (2026)
- [GenTe: Generative Real-world Terrains for General Legged Robot Locomotion Control](https://mlsanity.com/paper/2504.09997/) (2025)
- [CompSLAM: Complementary Hierarchical Multi-Modal Localization and Mapping for Robot Autonomy in Underground Environments](https://mlsanity.com/paper/2505.06483/) (2025)
- [Sapling-NeRF: Geo-Localised Sapling Reconstruction in Forests for Ecological Monitoring](https://mlsanity.com/paper/2602.22731/) (2026)
- [Sensor Configuration Matters: A Systematic Evaluation of Multimodal SLAM on Quadruped Robots](https://mlsanity.com/paper/2606.19067/) (2026)
- [Vernata: Self-Supervised Learning of LiDAR Point Representations](https://mlsanity.com/paper/2608.06919/) (2026)

## Abstract (arXiv)

We present visual inertial lidar legged navigation system (VILENS), an odometry system for legged robots based on factor graphs. The key novelty is the tight fusion of four different sensor modalities to achieve reliable operation when the individual sensors would otherwise produce degenerate estimation. To minimize leg odometry drift, we extend the robot's state with a linear velocity bias term, which is estimated online. This bias is observable because of the tight fusion of this preintegrated velocity factor with vision, lidar, and inertial measurement unit (IMU) factors. Extensive experimental validation on different ANYmal quadruped robots is presented, for a total duration of 2 h and 1.8 km traveled. The experiments involved dynamic locomotion over loose rocks, slopes, and mud, which caused challenges such as slippage and terrain deformation. Perceptual challenges included dark and dusty underground caverns, and open and feature-deprived areas. We show an average improvement of 62% translational and 51% rotational errors compared to a state-of-the-art loosely coupled approach. To demonstrate its robustness, VILENS was also integrated with a perceptive controller and a local path planner.