# MAGIC-VFM: Meta-learning Adaptation for Ground Interaction Control with Visual Foundation Models

arXiv: https://arxiv.org/abs/2407.12304 · HTML: https://mlsanity.com/paper/2407.12304/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2024.

## Built on by (most cited first, 6 of 6)

- [Morphological-Symmetry-Equivariant Heterogeneous Graph Neural Network for Robotic Dynamics Learning](https://mlsanity.com/paper/2412.01297/) (2024)
- [Meta-Learning Online Dynamics Model Adaptation in Off-Road Autonomous Driving](https://mlsanity.com/paper/2504.16923/) (2025)
- [Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving](https://mlsanity.com/paper/2412.00581/) (2024)
- [Data-Driven Modeling and Correction of Vehicle Dynamics](https://mlsanity.com/paper/2512.00289/) (2025)
- [The Open Ant: A Robot Platform for Reinforcement Learning Research](https://mlsanity.com/paper/2607.18488/) (2026)
- [Statistical Learning of Contractive Dynamical Representations for Composite Adaptive Control](https://mlsanity.com/paper/2609.35758/) (2026)

## Abstract (arXiv)

Control of off-road vehicles is challenging due to the complex dynamic interactions with the terrain. Accurate modeling of these interactions is important to optimize driving performance, but the relevant physical phenomena are too complex to model from first principles. Therefore, we present an offline meta-learning algorithm to construct a rapidly-tunable model of residual dynamics and disturbances. Our model processes terrain images into features using a visual foundation model (VFM), then maps these features and the vehicle state to an estimate of the current actuation matrix using a deep neural network (DNN). We then combine this model with composite adaptive control to modify the last layer of the DNN in real time, accounting for the remaining terrain interactions not captured during offline training. We provide mathematical guarantees of stability and robustness for our controller and demonstrate the effectiveness of our method through simulations and hardware experiments with a tracked vehicle and a car-like robot. We evaluate our method outdoors on different slopes with varying slippage and actuator degradation disturbances, and compare against an adaptive controller that does not use the VFM terrain features. We show significant improvement over the baseline in both hardware experimentation and simulation.