# VI-IKD: High-Speed Accurate Off-Road Navigation using Learned Visual-Inertial Inverse Kinodynamics

arXiv: https://arxiv.org/abs/2203.15983 · HTML: https://mlsanity.com/paper/2203.15983/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 35 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.07845/) (2016)
- [Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation](https://mlsanity.com/paper/1703.00420/) (2017)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [From Perception to Decision: A Data-driven Approach to End-to-end Motion Planning for Autonomous Ground Robots](https://mlsanity.com/paper/1609.07910/) (2016)
- [VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigation](https://mlsanity.com/paper/2105.09371/) (2021)
- [Learning Inverse Kinodynamics for Accurate High-Speed Off-Road Navigation on Unstructured Terrain](https://mlsanity.com/paper/2102.12667/) (2021)
- [Stochastic Grounded Action Transformation for Robot Learning in Simulation](https://mlsanity.com/paper/2008.01281/) (2020)
- [Toward Agile Maneuvers in Highly Constrained Spaces: Learning from Hallucination](https://mlsanity.com/paper/2007.14479/) (2020)
- [Learning to Drive Off Road on Smooth Terrain in Unstructured Environments Using an On-Board Camera and Sparse Aerial Images](https://mlsanity.com/paper/2004.04697/) (2020)
- [From Agile Ground to Aerial Navigation: Learning from Learned Hallucination](https://mlsanity.com/paper/2108.09793/) (2021)
- [Vision-Based High Speed Driving with a Deep Dynamic Observer](https://mlsanity.com/paper/1812.02071/) (2018)
- [Agile Robot Navigation through Hallucinated Learning and Sober Deployment](https://mlsanity.com/paper/2010.08098/) (2020)

## Built on by (most cited first, 21 of 21)

- [Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation](https://mlsanity.com/paper/2209.10780/) (2022)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [DTG : Diffusion-based Trajectory Generation for Mapless Global Navigation](https://mlsanity.com/paper/2403.09900/) (2024)
- [High-Speed Accurate Robot Control using Learned Forward Kinodynamics and Non-linear Least Squares Optimization](https://mlsanity.com/paper/2206.08487/) (2022)
- [STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience](https://mlsanity.com/paper/2309.15302/) (2023)
- [VINet: Visual and Inertial-based Terrain Classification and Adaptive Navigation over Unknown Terrain](https://mlsanity.com/paper/2209.07725/) (2022)
- [Learning Terrain-Aware Kinodynamic Model for Autonomous Off-Road Rally Driving With Model Predictive Path Integral Control](https://mlsanity.com/paper/2305.00676/) (2023)
- [Bridging Active Exploration and Uncertainty-Aware Deployment Using Probabilistic Ensemble Neural Network Dynamics](https://mlsanity.com/paper/2305.12240/) (2023)
- [Wait, That Feels Familiar: Learning to Extrapolate Human Preferences for Preference Aligned Path Planning](https://mlsanity.com/paper/2309.09912/) (2023)
- [Reinforcement Learning for Wheeled Mobility on Vertically Challenging Terrain](https://mlsanity.com/paper/2409.02383/) (2024)
- [SOCIALGYM 2.0: Simulator for Multi-Agent Social Robot Navigation in Shared Human Spaces](https://mlsanity.com/paper/2303.05584/) (2023)
- [VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Training](https://mlsanity.com/paper/2403.08109/) (2024)
- [Grounded Curriculum Learning](https://mlsanity.com/paper/2409.19816/) (2024)
- [A Study on Learning Social Robot Navigation with Multimodal Perception](https://mlsanity.com/paper/2309.12568/) (2023)
- [VertiSelector: Automatic Curriculum Learning for Wheeled Mobility on Vertically Challenging Terrain](https://mlsanity.com/paper/2409.17469/) (2024)
- [GACL: Grounded Adaptive Curriculum Learning with Active Task and Performance Monitoring](https://mlsanity.com/paper/2508.02988/) (2025)
- [WROOM: An Autonomous Driving Approach for Off-Road Navigation](https://mlsanity.com/paper/2404.08855/) (2024)
- [T-CBF: Traversability-based Control Barrier Function to Navigate Vertically Challenging Terrain](https://mlsanity.com/paper/2503.06083/) (2025)
- [Dyna-LfLH: Learning Agile Navigation in Dynamic Environments from Learned Hallucination](https://mlsanity.com/paper/2403.17231/) (2024)
- [Aligning Robot Navigation Behaviors with Human Intentions and Preferences](https://mlsanity.com/paper/2409.18982/) (2024)
- [Adapting Generalist Vehicle Models for High-Speed MPC Across Terrains](https://mlsanity.com/paper/2607.13319/) (2026)

## Abstract (arXiv)

One of the key challenges in high speed off road navigation on ground vehicles is that the kinodynamics of the vehicle terrain interaction can differ dramatically depending on the terrain. Previous approaches to addressing this challenge have considered learning an inverse kinodynamics (IKD) model, conditioned on inertial information of the vehicle to sense the kinodynamic interactions. In this paper, we hypothesize that to enable accurate high-speed off-road navigation using a learned IKD model, in addition to inertial information from the past, one must also anticipate the kinodynamic interactions of the vehicle with the terrain in the future. To this end, we introduce Visual-Inertial Inverse Kinodynamics (VI-IKD), a novel learning based IKD model that is conditioned on visual information from a terrain patch ahead of the robot in addition to past inertial information, enabling it to anticipate kinodynamic interactions in the future. We validate the effectiveness of VI-IKD in accurate high-speed off-road navigation experimentally on a scale 1/5 UT-AlphaTruck off-road autonomous vehicle in both indoor and outdoor environments and show that compared to other state-of-the-art approaches, VI-IKD enables more accurate and robust off-road navigation on a variety of different terrains at speeds of up to 3.5 m/s.