# Mars Traversability Prediction: A Multi-modal Self-supervised Approach for Costmap Generation

arXiv: https://arxiv.org/abs/2509.11082 · HTML: https://mlsanity.com/paper/2509.11082/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 14 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models](https://mlsanity.com/paper/2205.01791/) (2022)
- [Fast Traversability Estimation for Wild Visual Navigation](https://mlsanity.com/paper/2305.08510/) (2023)
- [FAST-LIVO2: Fast, Direct LiDAR-Inertial-Visual Odometry](https://mlsanity.com/paper/2408.14035/) (2024)
- [WayFAST: Navigation with Predictive Traversability in the Field](https://mlsanity.com/paper/2203.12071/) (2022)
- [V-STRONG: Visual Self-Supervised Traversability Learning for Off-road Navigation](https://mlsanity.com/paper/2312.16016/) (2023)
- [How Does It Feel? Self-Supervised Costmap Learning for Off-Road Vehicle Traversability](https://mlsanity.com/paper/2209.10788/) (2022)
- [Learning Off-Road Terrain Traversability with Self-Supervisions Only](https://mlsanity.com/paper/2305.18896/) (2023)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [Risk-Aware Off-Road Navigation via a Learned Speed Distribution Map](https://mlsanity.com/paper/2203.13429/) (2022)
- [Learning Risk-aware Costmaps for Traversability in Challenging Environments](https://mlsanity.com/paper/2107.11722/) (2021)
- [RCA: Ride Comfort-Aware Visual Navigation via Self-Supervised Learning](https://mlsanity.com/paper/2207.14460/) (2022)
- [Learning to Drive Off Road on Smooth Terrain in Unstructured Environments Using an On-Board Camera and Sparse Aerial Images](https://mlsanity.com/paper/2004.04697/) (2020)
- [Follow the Footprints: Self-supervised Traversability Estimation for Off-road Vehicle Navigation based on Geometric and Visual Cues](https://mlsanity.com/paper/2402.15363/) (2024)
- [MLNav: Learning to Safely Navigate on Martian Terrains](https://mlsanity.com/paper/2203.04563/) (2022)
- [Learning-based Traversability Costmap for Autonomous Off-road Navigation](https://mlsanity.com/paper/2406.08187/) (2024)

## Abstract (arXiv)

We present a robust multi-modal framework for predicting traversability costmaps for planetary rovers. Our model fuses camera and LiDAR data to produce a bird's-eye-view (BEV) terrain costmap, trained self-supervised using IMU-derived labels. Key updates include a DINOv3-based image encoder, FiLM-based sensor fusion, and an optimization loss combining Huber and smoothness terms. Experimental ablations (removing image color, occluding inputs, adding noise) show only minor changes in MAE/MSE (e.g. MAE increases from ~0.0775 to 0.0915 when LiDAR is sparsified), indicating that geometry dominates the learned cost and the model is highly robust. We attribute the small performance differences to the IMU labeling primarily reflecting terrain geometry rather than semantics and to limited data diversity. Unlike prior work claiming large gains, we emphasize our contributions: (1) a high-fidelity, reproducible simulation environment; (2) a self-supervised IMU-based labeling pipeline; and (3) a strong multi-modal BEV costmap prediction model. We discuss limitations and future work such as domain generalization and dataset expansion.