# TerrainNet: Visual Modeling of Complex Terrain for High-speed, Off-road Navigation

arXiv: https://arxiv.org/abs/2303.15771 · HTML: https://mlsanity.com/paper/2303.15771/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2023.

## Built on by (most cited first, 30 of 30)

- [TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation](https://mlsanity.com/paper/2505.10696/) (2025)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [EVORA: Deep Evidential Traversability Learning for Risk-Aware Off-Road Autonomy](https://mlsanity.com/paper/2311.06234/) (2023)
- [STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience](https://mlsanity.com/paper/2309.15302/) (2023)
- [Learning to See Physical Properties with Active Sensing Motor Policies](https://mlsanity.com/paper/2311.01405/) (2023)
- [Pixel to Elevation: Learning to Predict Elevation Maps at Long Range using Images for Autonomous Offroad Navigation](https://mlsanity.com/paper/2401.17484/) (2024)
- [CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidance](https://mlsanity.com/paper/2503.03921/) (2025)
- [PIETRA: Physics-Informed Evidential Learning for Traversing Out-of-Distribution Terrain](https://mlsanity.com/paper/2409.03005/) (2024)
- [Reinforcement Learning for Wheeled Mobility on Vertically Challenging Terrain](https://mlsanity.com/paper/2409.02383/) (2024)
- [VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation](https://mlsanity.com/paper/2510.20818/) (2025)
- [Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigation](https://mlsanity.com/paper/2410.10766/) (2024)
- [UFO: Uncertainty-aware LiDAR-image Fusion for Off-road Semantic Terrain Map Estimation](https://mlsanity.com/paper/2403.02642/) (2024)
- [Map It Anywhere (MIA): Empowering Bird's Eye View Mapping using Large-scale Public Data](https://mlsanity.com/paper/2407.08726/) (2024)
- [RoadRunner M&M -- Learning Multi-range Multi-resolution Traversability Maps for Autonomous Off-road Navigation](https://mlsanity.com/paper/2409.10940/) (2024)
- [PACER: Preference-conditioned All-terrain Costmap Generation](https://mlsanity.com/paper/2410.23488/) (2024)
- [Watch Your STEPP: Semantic Traversability Estimation using Pose Projected Features](https://mlsanity.com/paper/2501.17594/) (2025)
- [Deep Bayesian Future Fusion for Self-Supervised, High-Resolution, Off-Road Mapping](https://mlsanity.com/paper/2403.11876/) (2024)
- [VertiSelector: Automatic Curriculum Learning for Wheeled Mobility on Vertically Challenging Terrain](https://mlsanity.com/paper/2409.17469/) (2024)
- [Meta-Learning Online Dynamics Model Adaptation in Off-Road Autonomous Driving](https://mlsanity.com/paper/2504.16923/) (2025)
- [WildOS: Open-Vocabulary Object Search in the Wild](https://mlsanity.com/paper/2602.19308/) (2026)
- [TOP-Nav: Legged Navigation Integrating Terrain, Obstacle and Proprioception Estimation](https://mlsanity.com/paper/2404.15256/) (2024)
- [Anomalies by Synthesis: Anomaly Detection using Generative Diffusion Models for Off-Road Navigation](https://mlsanity.com/paper/2505.22805/) (2025)
- [Learning a Terrain- and Robot-Aware Dynamics Model for Autonomous Mobile Robot Navigation](https://mlsanity.com/paper/2409.11452/) (2024)
- [Temporally Consistent Unsupervised Segmentation for Mobile Robot Perception](https://mlsanity.com/paper/2507.22194/) (2025)
- [Lift, Splat, Map: Lifting Foundation Masks for Label-Free Semantic Scene Completion](https://mlsanity.com/paper/2407.03425/) (2024)
- [ADEPT: Adaptive Diffusion Environment for Policy Transfer Sim-to-Real](https://mlsanity.com/paper/2506.01759/) (2025)
- [Learning Smooth State-Dependent Traversability from Dense Point Clouds](https://mlsanity.com/paper/2506.04362/) (2025)
- [OT-Drive: Out-of-Distribution Off-Road Traversable Area Segmentation via Optimal Transport](https://mlsanity.com/paper/2601.09952/) (2026)
- [Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs](https://mlsanity.com/paper/2604.04564/) (2026)
- [Multimodal embodiment-aware navigation transformer](https://mlsanity.com/paper/2604.19267/) (2026)

## Abstract (arXiv)

Effective use of camera-based vision systems is essential for robust performance in autonomous off-road driving, particularly in the high-speed regime. Despite success in structured, on-road settings, current end-to-end approaches for scene prediction have yet to be successfully adapted for complex outdoor terrain. To this end, we present TerrainNet, a vision-based terrain perception system for semantic and geometric terrain prediction for aggressive, off-road navigation. The approach relies on several key insights and practical considerations for achieving reliable terrain modeling. The network includes a multi-headed output representation to capture fine- and coarse-grained terrain features necessary for estimating traversability. Accurate depth estimation is achieved using self-supervised depth completion with multi-view RGB and stereo inputs. Requirements for real-time performance and fast inference speeds are met using efficient, learned image feature projections. Furthermore, the model is trained on a large-scale, real-world off-road dataset collected across a variety of diverse outdoor environments. We show how TerrainNet can also be used for costmap prediction and provide a detailed framework for integration into a planning module. We demonstrate the performance of TerrainNet through extensive comparison to current state-of-the-art baselines for camera-only scene prediction. Finally, we showcase the effectiveness of integrating TerrainNet within a complete autonomous-driving stack by conducting a real-world vehicle test in a challenging off-road scenario.