# RoadFormer+: Delivering RGB-X Scene Parsing through Scale-Aware Information Decoupling and Advanced Heterogeneous Feature Fusion

arXiv: https://arxiv.org/abs/2407.21631 · HTML: https://mlsanity.com/paper/2407.21631/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 78 references that are in the map.
- First posted on arXiv 31 Jul 2024.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [Coordinate Attention for Efficient Mobile Network Design](https://mlsanity.com/paper/2103.02907/) (2021)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation](https://mlsanity.com/paper/2308.02097/) (2023)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes](https://mlsanity.com/paper/1708.01566/) (2017)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)

## Built on by (most cited first, 10 of 10)

- [CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes](https://mlsanity.com/paper/2410.10791/) (2024)
- [Depth Matters: Exploring Deep Interactions of RGB-D for Semantic Segmentation in Traffic Scenes](https://mlsanity.com/paper/2409.07995/) (2024)
- [Multimodal SAM-adapter for Semantic Segmentation](https://mlsanity.com/paper/2509.10408/) (2025)
- [Fully Exploiting Vision Foundation Model's Profound Prior Knowledge for Generalizable RGB-Depth Driving Scene Parsing](https://mlsanity.com/paper/2502.06219/) (2025)
- [DepthMatch: Semi-Supervised RGB-D Scene Parsing through Depth-Guided Regularization](https://mlsanity.com/paper/2505.20041/) (2025)
- [MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation](https://mlsanity.com/paper/2509.11817/) (2025)
- [Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks](https://mlsanity.com/paper/2602.13588/) (2026)
- [Structured prototype regularization for synthetic-to-real driving scene parsing](https://mlsanity.com/paper/2603.16083/) (2026)
- [IAF-Net: Illumination-Adaptive Fusion for Low-Light Urban Road Segmentation](https://mlsanity.com/paper/2605.30939/) (2026)
- [Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation](https://mlsanity.com/paper/2610.17800/) (2026)

## Abstract (arXiv)

Task-specific data-fusion networks have marked considerable achievements in urban scene parsing. Among these networks, our recently proposed RoadFormer successfully extracts heterogeneous features from RGB images and surface normal maps and fuses these features through attention mechanisms, demonstrating compelling efficacy in RGB-Normal road scene parsing. However, its performance significantly deteriorates when handling other types/sources of data or performing more universal, all-category scene parsing tasks. To overcome these limitations, this study introduces RoadFormer+, an efficient, robust, and adaptable model capable of effectively fusing RGB-X data, where ``X'', represents additional types/modalities of data such as depth, thermal, surface normal, and polarization. Specifically, we propose a novel hybrid feature decoupling encoder to extract heterogeneous features and decouple them into global and local components. These decoupled features are then fused through a dual-branch multi-scale heterogeneous feature fusion block, which employs parallel Transformer attentions and convolutional neural network modules to merge multi-scale features across different scales and receptive fields. The fused features are subsequently fed into a decoder to generate the final semantic predictions. Notably, our proposed RoadFormer+ ranks first on the KITTI Road benchmark and achieves state-of-the-art performance in mean intersection over union on the Cityscapes, MFNet, FMB, and ZJU datasets. Moreover, it reduces the number of learnable parameters by 65% compared to RoadFormer. Our source code will be publicly available at mias.group/RoadFormerPlus.