# Few-shot Semantic Learning for Robust Multi-Biome 3D Semantic Mapping in Off-Road Environments

arXiv: https://arxiv.org/abs/2411.06632 · HTML: https://mlsanity.com/paper/2411.06632/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 10 Nov 2024.

## Builds on (most cited first, 15 of 15)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning](https://mlsanity.com/paper/1611.03631/) (2016)
- [PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things](https://mlsanity.com/paper/1903.01177/) (2019)
- [Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery](https://mlsanity.com/paper/1903.00268/) (2019)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [Self-Supervised Traversability Prediction by Learning to Reconstruct Safe Terrain](https://mlsanity.com/paper/2208.01329/) (2022)
- [Learning-on-the-Drive: Self-supervised Adaptation of Visual Offroad Traversability Models](https://mlsanity.com/paper/2306.15226/) (2023)
- [Pixel to Elevation: Learning to Predict Elevation Maps at Long Range using Images for Autonomous Offroad Navigation](https://mlsanity.com/paper/2401.17484/) (2024)
- [G-VOM: A GPU Accelerated Voxel Off-Road Mapping System](https://mlsanity.com/paper/2109.13176/) (2021)
- [Robust Fusion for Bayesian Semantic Mapping](https://mlsanity.com/paper/2303.07836/) (2023)
- [RoadRunner M&M -- Learning Multi-range Multi-resolution Traversability Maps for Autonomous Off-road Navigation](https://mlsanity.com/paper/2409.10940/) (2024)
- [ROAMER: Robust Offroad Autonomy using Multimodal State Estimation with Radar Velocity Integration](https://mlsanity.com/paper/2401.17404/) (2024)

## Built on by (most cited first, 3 of 3)

- [Meta-Learning Online Dynamics Model Adaptation in Off-Road Autonomous Driving](https://mlsanity.com/paper/2504.16923/) (2025)
- [Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving](https://mlsanity.com/paper/2412.00581/) (2024)
- [COARSE: Collaborative Pseudo-Labeling with Coarse Real Labels for Off-Road Semantic Segmentation](https://mlsanity.com/paper/2503.03947/) (2025)

## Abstract (arXiv)

Off-road environments pose significant perception challenges for high-speed autonomous navigation due to unstructured terrain, degraded sensing conditions, and domain-shifts among biomes. Learning semantic information across these conditions and biomes can be challenging when a large amount of ground truth data is required. In this work, we propose an approach that leverages a pre-trained Vision Transformer (ViT) with fine-tuning on a small (<500 images), sparse and coarsely labeled (<30% pixels) multi-biome dataset to predict 2D semantic segmentation classes. These classes are fused over time via a novel range-based metric and aggregated into a 3D semantic voxel map. We demonstrate zero-shot out-of-biome 2D semantic segmentation on the Yamaha (52.9 mIoU) and Rellis (55.5 mIoU) datasets along with few-shot coarse sparse labeling with existing data for improved segmentation performance on Yamaha (66.6 mIoU) and Rellis (67.2 mIoU). We further illustrate the feasibility of using a voxel map with a range-based semantic fusion approach to handle common off-road hazards like pop-up hazards, overhangs, and water features.