# A Cross-Scale Decoder with Token Refinement for Off-Road Semantic Segmentation

arXiv: https://arxiv.org/abs/2603.27931 · HTML: https://mlsanity.com/paper/2603.27931/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://mlsanity.com/paper/2004.02147/) (2020)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [Boundary loss for highly unbalanced segmentation](https://mlsanity.com/paper/1812.07032/) (2018)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [Gated-SCNN: Gated Shape CNNs for Semantic Segmentation](https://mlsanity.com/paper/1907.05740/) (2019)
- [Fast-SCNN: Fast Semantic Segmentation Network](https://mlsanity.com/paper/1902.04502/) (2019)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [Boundary-Aware Feature Propagation for Scene Segmentation](https://mlsanity.com/paper/1909.00179/) (2019)
- [SegFix: Model-Agnostic Boundary Refinement for Segmentation](https://mlsanity.com/paper/2007.04269/) (2020)
- [CGNet: A Light-weight Context Guided Network for Semantic Segmentation](https://mlsanity.com/paper/1811.08201/) (2018)
- [FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation](https://mlsanity.com/paper/1903.11816/) (2019)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models](https://mlsanity.com/paper/2205.01791/) (2022)

## Abstract (arXiv)

Off-road semantic segmentation is fundamentally challenged by irregular terrain, vegetation clutter, and inherent annotation ambiguity. Unlike urban scenes with crisp object boundaries, off-road environments exhibit strong class-level similarity among terrain categories, resulting in thick and uncertain transition regions that degrade boundary coherence and destabilize training. Rare or thin structures, such as narrow traversable gaps or isolated obstacles, further receive sparse and unreliable supervision and are easily overwhelmed by dominant background textures. Existing decoder designs either rely on low-scale bottlenecks that oversmooth fine structural details, or repeatedly fuse high-detail features, which tends to amplify annotation noise and incur substantial computational cost. We present a cross-scale decoder that explicitly addresses these challenges through three complementary mechanisms. First, a global--local token refinement module consolidates semantic context on a compact bottleneck lattice, guided by boundary-aware regularization to remain robust under ambiguous supervision. Second, a gated detail bridge selectively injects fine-scale structural cues only once through cross-scale attention, preserving boundary and texture information while avoiding noise accumulation. Third, an uncertainty-guided class-aware point refinement selectively updates the least reliable pixels, improving rare and ambiguous structures with minimal computational overhead. The resulting framework achieves noise-robust and boundary-preserving segmentation tailored to off-road environments, recovering fine structural details while maintaining deployment-friendly efficiency. Experimental results on standard off-road benchmarks demonstrate consistent improvements over prior approaches without resorting to heavy dense feature fusion.