# Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving

arXiv: https://arxiv.org/abs/2412.00581 · HTML: https://mlsanity.com/paper/2412.00581/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 30 Nov 2024.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 15 of 15)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [Model Predictive Path Integral Control using Covariance Variable Importance Sampling](https://mlsanity.com/paper/1509.01149/) (2015)
- [Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving](https://mlsanity.com/paper/1707.02342/) (2017)
- [WayFAST: Navigation with Predictive Traversability in the Field](https://mlsanity.com/paper/2203.12071/) (2022)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [EVORA: Deep Evidential Traversability Learning for Risk-Aware Off-Road Autonomy](https://mlsanity.com/paper/2311.06234/) (2023)
- [Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision](https://mlsanity.com/paper/2404.07110/) (2024)
- [MEM: Multi-Modal Elevation Mapping for Robotics and Learning](https://mlsanity.com/paper/2309.16818/) (2023)
- [Identifying Terrain Physical Parameters from Vision -- Towards Physical-Parameter-Aware Locomotion and Navigation](https://mlsanity.com/paper/2408.16567/) (2024)
- [A Multi-step Dynamics Modeling Framework For Autonomous Driving In Multiple Environments](https://mlsanity.com/paper/2305.02241/) (2023)
- [MAGIC-VFM: Meta-learning Adaptation for Ground Interaction Control with Visual Foundation Models](https://mlsanity.com/paper/2407.12304/) (2024)
- [Contrastive Label Disambiguation for Self-Supervised Terrain Traversability Learning in Off-Road Environments](https://mlsanity.com/paper/2307.02871/) (2023)
- [Low Frequency Sampling in Model Predictive Path Integral Control](https://mlsanity.com/paper/2404.03094/) (2024)
- [Few-shot Semantic Learning for Robust Multi-Biome 3D Semantic Mapping in Off-Road Environments](https://mlsanity.com/paper/2411.06632/) (2024)

## Built on by (most cited first, 2 of 2)

- [Meta-Learning Online Dynamics Model Adaptation in Off-Road Autonomous Driving](https://mlsanity.com/paper/2504.16923/) (2025)
- [Verti-WM: A Physics-Aided Exteroceptive World Model for Off-Road Reinforcement Learning](https://mlsanity.com/paper/2609.23118/) (2026)

## Abstract (arXiv)

Rapid autonomous traversal of unstructured terrain is essential for scenarios such as disaster response, search and rescue, or planetary exploration. As a vehicle navigates at the limit of its capabilities over extreme terrain, its dynamics can change suddenly and dramatically. For example, high-speed and varying terrain can affect parameters such as traction, tire slip, and rolling resistance. To achieve effective planning in such environments, it is crucial to have a dynamics model that can accurately anticipate these conditions. In this work, we present a hybrid model that predicts the changing dynamics induced by the terrain as a function of visual inputs. We leverage a pre-trained visual foundation model (VFM) DINOv2, which provides rich features that encode fine-grained semantic information. To use this dynamics model for planning, we propose an end-to-end training architecture for a projection distance independent feature encoder that compresses the information from the VFM, enabling the creation of a lightweight map of the environment at runtime. We validate our architecture on an extensive dataset (hundreds of kilometers of aggressive off-road driving) collected across multiple locations as part of the DARPA Robotic Autonomy in Complex Environments with Resiliency (RACER) program. https://www.youtube.com/watch?v=dycTXxEosMk