# Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarks

arXiv: https://arxiv.org/abs/2510.16500 · HTML: https://mlsanity.com/paper/2510.16500/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 18 Oct 2025.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 30 of 32)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2304.05316/) (2023)
- [KISS-ICP: In Defense of Point-to-Point ICP -- Simple, Accurate, and Robust Registration If Done the Right Way](https://mlsanity.com/paper/2209.15397/) (2022)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.15208/) (2024)
- [The GOOSE Dataset for Perception in Unstructured Environments](https://mlsanity.com/paper/2310.16788/) (2023)
- [ORFD: A Dataset and Benchmark for Off-Road Freespace Detection](https://mlsanity.com/paper/2206.09907/) (2022)
- [V-STRONG: Visual Self-Supervised Traversability Learning for Off-road Navigation](https://mlsanity.com/paper/2312.16016/) (2023)
- [Learning Off-Road Terrain Traversability with Self-Supervisions Only](https://mlsanity.com/paper/2305.18896/) (2023)
- [DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model](https://mlsanity.com/paper/2410.10738/) (2024)
- [BotanicGarden: A High-Quality Dataset for Robot Navigation in Unstructured Natural Environments](https://mlsanity.com/paper/2306.14137/) (2023)
- [OFFSEG: A Semantic Segmentation Framework For Off-Road Driving](https://mlsanity.com/paper/2103.12417/) (2021)
- [Toward Wheeled Mobility on Vertically Challenging Terrain: Platforms, Datasets, and Algorithms](https://mlsanity.com/paper/2303.00998/) (2023)
- [WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments](https://mlsanity.com/paper/2312.15364/) (2023)
- [TNS: Terrain Traversability Mapping and Navigation System for Autonomous Excavators](https://mlsanity.com/paper/2109.06250/) (2021)
- [Pixel to Elevation: Learning to Predict Elevation Maps at Long Range using Images for Autonomous Offroad Navigation](https://mlsanity.com/paper/2401.17484/) (2024)
- [WildOcc: A Benchmark for Off-Road 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2410.15792/) (2024)
- [Fine-Grained Off-Road Semantic Segmentation and Mapping via Contrastive Learning](https://mlsanity.com/paper/2103.03651/) (2021)
- [VINet: Visual and Inertial-based Terrain Classification and Adaptive Navigation over Unknown Terrain](https://mlsanity.com/paper/2209.07725/) (2022)
- [Autonomous Driving in Unstructured Environments: How Far Have We Come?](https://mlsanity.com/paper/2410.07701/) (2024)
- [Trajectory Prediction for Autonomous Driving with Topometric Map](https://mlsanity.com/paper/2105.03869/) (2021)
- [OpenOcc: Open Vocabulary 3D Scene Reconstruction via Occupancy Representation](https://mlsanity.com/paper/2403.11796/) (2024)
- [UFO: Uncertainty-aware LiDAR-image Fusion for Off-road Semantic Terrain Map Estimation](https://mlsanity.com/paper/2403.02642/) (2024)
- [ROD: RGB-Only Fast and Efficient Off-road Freespace Detection](https://mlsanity.com/paper/2508.08697/) (2025)
- [OffRoadTranSeg: Semi-Supervised Segmentation using Transformers on OffRoad environments](https://mlsanity.com/paper/2106.13963/) (2021)
- [Self-Supervised 3D Traversability Estimation with Proxy Bank Guidance](https://mlsanity.com/paper/2211.11201/) (2022)

## Built on by (most cited first, 8 of 8)

- [OneOcc: Semantic Occupancy Prediction for Legged Robots with a Single Panoramic Camera](https://mlsanity.com/paper/2511.03571/) (2025)
- [Panoramic Multimodal Semantic Occupancy Prediction for Quadruped Robots](https://mlsanity.com/paper/2603.13108/) (2026)
- [Ground4D: Spatially-Grounded Feedforward 4D Reconstruction for Unstructured Off-Road Scenes](https://mlsanity.com/paper/2605.04435/) (2026)
- [Humanoid-OmniOcc: Stereo-Based Full-View Occupancy Dataset for Embodied AI](https://mlsanity.com/paper/2606.22971/) (2026)
- [OT-Drive: Out-of-Distribution Off-Road Traversable Area Segmentation via Optimal Transport](https://mlsanity.com/paper/2601.09952/) (2026)
- [Wild-Drive: Off-Road Scene Captioning and Path Planning via Robust Multi-modal Routing and Efficient Large Language Model](https://mlsanity.com/paper/2603.00694/) (2026)
- [Towards All-Day Perception for Off-Road Driving: A Large-Scale Multispectral Dataset and Comprehensive Benchmark](https://mlsanity.com/paper/2604.27499/) (2026)
- [Which Terrain Is Better? Preference Learning with VLM Prototypes for Off-Road Traversability Ranking](https://mlsanity.com/paper/2609.23673/) (2026)

## Abstract (arXiv)

A major bottleneck in off-road autonomous driving research lies in the scarcity of large-scale, high-quality datasets and benchmarks. To bridge this gap, we present ORAD-3D, which, to the best of our knowledge, is the largest dataset specifically curated for off-road autonomous driving. ORAD-3D covers a wide spectrum of terrains, including woodlands, farmlands, grasslands, riversides, gravel roads, cement roads, and rural areas, while capturing diverse environmental variations across weather conditions (sunny, rainy, foggy, and snowy) and illumination levels (bright daylight, daytime, twilight, and nighttime). Building upon this dataset, we establish a comprehensive suite of benchmark evaluations spanning five fundamental tasks: 2D free-space detection, 3D occupancy prediction, rough GPS-guided path planning, vision-language model-driven autonomous driving, and world model for off-road environments. Together, the dataset and benchmarks provide a unified and robust resource for advancing perception and planning in challenging off-road scenarios. The dataset and code will be made publicly available at https://github.com/chaytonmin/ORAD-3D.