# Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data

arXiv: https://arxiv.org/abs/2605.27644 · HTML: https://mlsanity.com/paper/2605.27644/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 26 May 2026.

## Builds on (most cited first, 24 of 24)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detection](https://mlsanity.com/paper/2211.11317/) (2022)
- [Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2012.06575/) (2020)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)
- [Semantic Segmentation using Vision Transformers: A survey](https://mlsanity.com/paper/2305.03273/) (2023)
- [Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift](https://mlsanity.com/paper/1908.01098/) (2019)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [Fast Traversability Estimation for Wild Visual Navigation](https://mlsanity.com/paper/2305.08510/) (2023)
- [WayFAST: Navigation with Predictive Traversability in the Field](https://mlsanity.com/paper/2203.12071/) (2022)
- [V-STRONG: Visual Self-Supervised Traversability Learning for Off-road Navigation](https://mlsanity.com/paper/2312.16016/) (2023)
- [Detecting Road Obstacles by Erasing Them](https://mlsanity.com/paper/2012.13633/) (2020)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Open-World Semantic Segmentation Including Class Similarity](https://mlsanity.com/paper/2403.07532/) (2024)
- [Bayesian Nonparametric Submodular Video Partition for Robust Anomaly Detection](https://mlsanity.com/paper/2203.12840/) (2022)
- [Learning Ground Traversability from Simulations](https://mlsanity.com/paper/1709.05368/) (2017)
- [Mars Terrain Segmentation with Less Labels](https://mlsanity.com/paper/2202.00791/) (2022)
- [DeepTerramechanics: Terrain Classification and Slip Estimation for Ground Robots via Deep Learning](https://mlsanity.com/paper/1806.07379/) (2018)
- [S5Mars: Semi-Supervised Learning for Mars Semantic Segmentation](https://mlsanity.com/paper/2207.01200/) (2022)
- [SCIM: Simultaneous Clustering, Inference, and Mapping for Open-World Semantic Scene Understanding](https://mlsanity.com/paper/2206.10670/) (2022)
- [TerraPN: Unstructured Terrain Navigation using Online Self-Supervised Learning](https://mlsanity.com/paper/2202.12873/) (2022)
- [Temporally Consistent Unsupervised Segmentation for Mobile Robot Perception](https://mlsanity.com/paper/2507.22194/) (2025)

## Abstract (arXiv)

Terrain understanding is fundamental for mobile robots operating in unstructured outdoor environments. Existing vision-based traversability estimation methods rely on robot-specific annotations or semantic class mappings, limiting transferability across platforms and requiring costly re-annotation when robot capabilities change, while standard semantic segmentation methods only focus on specific predefined classes, which do not capture the variety of terrains. In this work, we propose a transformer-based architecture that jointly performs class-specific semantic segmentation and class-agnostic terrain segmentation within a unified network, called Trinity. Terrain regions are segmented based solely on visual appearance, without predefined semantic labels or robot-dependent traversability scores. This formulation enables the learning of robot-agnostic visual terrain priors that can be combined with robot-specific experience for downstream tasks such as traversability estimation, visual odometry, and mission planning. To enable large-scale training with diverse terrain appearances, we extend the OAISYS simulator and introduce RUGDSynth, a synthetic dataset inspired by RUGD with class-agnostic terrain samples. Furthermore, we present the EXTerra Dataset, providing real-world images annotated with both class-specific and class-agnostic terrain labels. Experiments demonstrate the feasibility of the proposed task and the effectiveness of our joint segmentation approach in complex outdoor environments. Code and datasets will be released with this publication (after review).