# RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving

arXiv: https://arxiv.org/abs/2402.19341 · HTML: https://mlsanity.com/paper/2402.19341/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 56 of its 105 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 30 of 56)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Orthographic Feature Transform for Monocular 3D Object Detection](https://mlsanity.com/paper/1811.08188/) (2018)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation](https://mlsanity.com/paper/2204.05088/) (2022)
- [Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?](https://mlsanity.com/paper/2206.07959/) (2022)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)
- [Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving](https://mlsanity.com/paper/1707.02342/) (2017)
- [How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics](https://mlsanity.com/paper/1710.11040/) (2017)
- [ViPlanner: Visual Semantic Imperative Learning for Local Navigation](https://mlsanity.com/paper/2310.00982/) (2023)
- [Fast Traversability Estimation for Wild Visual Navigation](https://mlsanity.com/paper/2305.08510/) (2023)
- [Self-Supervised Visual Terrain Classification from Unsupervised Acoustic Feature Learning](https://mlsanity.com/paper/1912.03227/) (2019)
- [BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud](https://mlsanity.com/paper/2006.11436/) (2020)
- [Safe Robot Navigation via Multi-Modal Anomaly Detection](https://mlsanity.com/paper/2001.07934/) (2020)
- [WayFAST: Navigation with Predictive Traversability in the Field](https://mlsanity.com/paper/2203.12071/) (2022)
- [V-STRONG: Visual Self-Supervised Traversability Learning for Off-road Navigation](https://mlsanity.com/paper/2312.16016/) (2023)
- [TerrainNet: Visual Modeling of Complex Terrain for High-speed, Off-road Navigation](https://mlsanity.com/paper/2303.15771/) (2023)
- [Locomotion Policy Guided Traversability Learning using Volumetric Representations of Complex Environments](https://mlsanity.com/paper/2203.15854/) (2022)
- [STEP: Stochastic Traversability Evaluation and Planning for Risk-Aware Off-road Navigation](https://mlsanity.com/paper/2103.02828/) (2021)
- [Autonomous Exploration Development Environment and the Planning Algorithms](https://mlsanity.com/paper/2110.14573/) (2021)
- [How Does It Feel? Self-Supervised Costmap Learning for Off-Road Vehicle Traversability](https://mlsanity.com/paper/2209.10788/) (2022)
- [Learning Off-Road Terrain Traversability with Self-Supervisions Only](https://mlsanity.com/paper/2305.18896/) (2023)
- [Learning Inverse Kinodynamics for Accurate High-Speed Off-Road Navigation on Unstructured Terrain](https://mlsanity.com/paper/2102.12667/) (2021)

## Built on by (most cited first, 26 of 26)

- [TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation](https://mlsanity.com/paper/2505.10696/) (2025)
- [Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environments](https://mlsanity.com/paper/2409.20445/) (2024)
- [PIETRA: Physics-Informed Evidential Learning for Traversing Out-of-Distribution Terrain](https://mlsanity.com/paper/2409.03005/) (2024)
- [Autonomous Driving in Unstructured Environments: How Far Have We Come?](https://mlsanity.com/paper/2410.07701/) (2024)
- [PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation](https://mlsanity.com/paper/2511.18570/) (2025)
- [RoadRunner M&M -- Learning Multi-range Multi-resolution Traversability Maps for Autonomous Off-road Navigation](https://mlsanity.com/paper/2409.10940/) (2024)
- [Watch Your STEPP: Semantic Traversability Estimation using Pose Projected Features](https://mlsanity.com/paper/2501.17594/) (2025)
- [Deep Bayesian Future Fusion for Self-Supervised, High-Resolution, Off-Road Mapping](https://mlsanity.com/paper/2403.11876/) (2024)
- [Few-shot Semantic Learning for Robust Multi-Biome 3D Semantic Mapping in Off-Road Environments](https://mlsanity.com/paper/2411.06632/) (2024)
- [Wheeled Lab: Modern Sim2Real for Low-cost, Open-source Wheeled Robotics](https://mlsanity.com/paper/2502.07380/) (2025)
- [WildOS: Open-Vocabulary Object Search in the Wild](https://mlsanity.com/paper/2602.19308/) (2026)
- [TADPO: Reinforcement Learning Goes Off-road](https://mlsanity.com/paper/2603.05995/) (2026)
- [TOP-Nav: Legged Navigation Integrating Terrain, Obstacle and Proprioception Estimation](https://mlsanity.com/paper/2404.15256/) (2024)
- [Uncertainty-aware Semantic Mapping in Off-road Environments with Dempster-Shafer Theory of Evidence](https://mlsanity.com/paper/2405.06265/) (2024)
- [Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving](https://mlsanity.com/paper/2412.00581/) (2024)
- [Trajectory-based Road Autolabeling with Lidar-Camera Fusion in Winter Conditions](https://mlsanity.com/paper/2412.02370/) (2024)
- [FusionForce: End-to-end Differentiable Neural-Symbolic Layer for Trajectory Prediction](https://mlsanity.com/paper/2502.10156/) (2025)
- [Self-supervised cost of transport estimation for multimodal path planning](https://mlsanity.com/paper/2412.06101/) (2024)
- [Holistic Fusion: Task- and Setup-Agnostic Robot Localization and State Estimation with Factor Graphs](https://mlsanity.com/paper/2504.06479/) (2025)
- [Learning Smooth State-Dependent Traversability from Dense Point Clouds](https://mlsanity.com/paper/2506.04362/) (2025)
- [Experimental investigation of pose informed reinforcement learning for skid-steered visual navigation](https://mlsanity.com/paper/2506.21732/) (2025)
- [Mars Traversability Prediction: A Multi-modal Self-supervised Approach for Costmap Generation](https://mlsanity.com/paper/2509.11082/) (2025)
- [Splatblox: Traversability-Aware Gaussian Splatting for Outdoor Robot Navigation](https://mlsanity.com/paper/2511.18525/) (2025)
- [Multimodal embodiment-aware navigation transformer](https://mlsanity.com/paper/2604.19267/) (2026)
- [From Language Priors to Field Adaptation: Preference Learning for Traversability Estimation](https://mlsanity.com/paper/2610.02974/) (2026)
- [Robust 2D Traversability Mapping for Construction AMRs via Failure-Mode-Aware Fusion of LiDAR Geometry and Monocular Semantics](https://mlsanity.com/paper/2610.05505/) (2026)

## Abstract (arXiv)

Autonomous navigation at high speeds in off-road environments necessitates robots to comprehensively understand their surroundings using onboard sensing only. The extreme conditions posed by the off-road setting can cause degraded camera image quality due to poor lighting and motion blur, as well as limited sparse geometric information available from LiDAR sensing when driving at high speeds. In this work, we present RoadRunner, a novel framework capable of predicting terrain traversability and an elevation map directly from camera and LiDAR sensor inputs. RoadRunner enables reliable autonomous navigation, by fusing sensory information, handling of uncertainty, and generation of contextually informed predictions about the geometry and traversability of the terrain while operating at low latency. In contrast to existing methods relying on classifying handcrafted semantic classes and using heuristics to predict traversability costs, our method is trained end-to-end in a self-supervised fashion. The RoadRunner network architecture builds upon popular sensor fusion network architectures from the autonomous driving domain, which embed LiDAR and camera information into a common Bird's Eye View perspective. Training is enabled by utilizing an existing traversability estimation stack to generate training data in hindsight in a scalable manner from real-world off-road driving datasets. Furthermore, RoadRunner improves the system latency by a factor of roughly 4, from 500 ms to 140 ms, while improving the accuracy for traversability costs and elevation map predictions. We demonstrate the effectiveness of RoadRunner in enabling safe and reliable off-road navigation at high speeds in multiple real-world driving scenarios through unstructured desert environments.