# VINet: Visual and Inertial-based Terrain Classification and Adaptive Navigation over Unknown Terrain

arXiv: https://arxiv.org/abs/2209.07725 · HTML: https://mlsanity.com/paper/2209.07725/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [Data-Driven MPC for Quadrotors](https://mlsanity.com/paper/2102.05773/) (2021)
- [Deep Texture Manifold for Ground Terrain Recognition](https://mlsanity.com/paper/1803.10896/) (2018)
- [DenseCAvoid: Real-time Navigation in Dense Crowds using Anticipatory Behaviors](https://mlsanity.com/paper/2002.03038/) (2020)
- [GrASPE: Graph based Multimodal Fusion for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2209.05722/) (2022)
- [VI-IKD: High-Speed Accurate Off-Road Navigation using Learned Visual-Inertial Inverse Kinodynamics](https://mlsanity.com/paper/2203.15983/) (2022)
- [OFFSEG: A Semantic Segmentation Framework For Off-Road Driving](https://mlsanity.com/paper/2103.12417/) (2021)
- [TERP: Reliable Planning in Uneven Outdoor Environments using Deep Reinforcement Learning](https://mlsanity.com/paper/2109.05120/) (2021)
- [Frozone: Freezing-Free, Pedestrian-Friendly Navigation in Human Crowds](https://mlsanity.com/paper/2003.05395/) (2020)
- [TNS: Terrain Traversability Mapping and Navigation System for Autonomous Excavators](https://mlsanity.com/paper/2109.06250/) (2021)
- [Differential Angular Imaging for Material Recognition](https://mlsanity.com/paper/1612.02372/) (2016)
- [Learn Fast, Forget Slow: Safe Predictive Learning Control for Systems with Unknown and Changing Dynamics Performing Repetitive Tasks](https://mlsanity.com/paper/1810.06681/) (2018)
- [Surface Type Classification for Autonomous Robot Indoor Navigation](https://mlsanity.com/paper/1905.00252/) (2019)
- [AdaptiveON: Adaptive Outdoor Local Navigation Method For Stable and Reliable Actions](https://mlsanity.com/paper/2205.03517/) (2022)
- [OffRoadTranSeg: Semi-Supervised Segmentation using Transformers on OffRoad environments](https://mlsanity.com/paper/2106.13963/) (2021)

## Built on by (most cited first, 8 of 8)

- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarks](https://mlsanity.com/paper/2510.16500/) (2025)
- [Autonomous Driving in Unstructured Environments: How Far Have We Come?](https://mlsanity.com/paper/2410.07701/) (2024)
- [QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds](https://mlsanity.com/paper/2406.16578/) (2024)
- [AGL-NET: Aerial-Ground Cross-Modal Global Localization with Varying Scales](https://mlsanity.com/paper/2404.03187/) (2024)
- [METAVerse: Meta-Learning Traversability Cost Map for Off-Road Navigation](https://mlsanity.com/paper/2307.13991/) (2023)
- [Leveraging Energy Features for Surface Classification with Deep Learning: A Comparative Analysis Across Three Independent Datasets](https://mlsanity.com/paper/2606.18698/) (2026)
- [Attention-Based Surface Representation Learning for Robot State Prediction and Open-Ended Surface Classification](https://mlsanity.com/paper/2610.04240/) (2026)

## Abstract (arXiv)

We present a visual and inertial-based terrain classification network (VINet) for robotic navigation over different traversable surfaces. We use a novel navigation-based labeling scheme for terrain classification and generalization on unknown surfaces. Our proposed perception method and adaptive scheduling control framework can make predictions according to terrain navigation properties and lead to better performance on both terrain classification and navigation control on known and unknown surfaces. Our VINet can achieve 98.37% in terms of accuracy under supervised setting on known terrains and improve the accuracy by 8.51% on unknown terrains compared to previous methods. We deploy VINet on a mobile tracked robot for trajectory following and navigation on different terrains, and we demonstrate an improvement of 10.3% compared to a baseline controller in terms of RMSE.