# CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation

arXiv: https://arxiv.org/abs/2604.02948 · HTML: https://mlsanity.com/paper/2604.02948/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 60 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Builds on (most cited first, 30 of 42)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [OCNet: Object Context Network for Scene Parsing](https://mlsanity.com/paper/1809.00916/) (2018)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [Strip Pooling: Rethinking Spatial Pooling for Scene Parsing](https://mlsanity.com/paper/2003.13328/) (2020)
- [Depth-aware CNN for RGB-D Segmentation](https://mlsanity.com/paper/1803.06791/) (2018)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)
- [Bi-directional Adapter for Multi-modal Tracking](https://mlsanity.com/paper/2312.10611/) (2023)
- [Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsing](https://mlsanity.com/paper/2112.05144/) (2021)
- [GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer](https://mlsanity.com/paper/2406.01210/) (2024)
- [StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation](https://mlsanity.com/paper/2408.01343/) (2024)
- [Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments](https://mlsanity.com/paper/2207.04526/) (2022)
- [MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty](https://mlsanity.com/paper/2401.12761/) (2024)
- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)

## Abstract (arXiv)

Multimodal semantic segmentation has shown great potential in leveraging complementary information across diverse sensing modalities. However, existing approaches often rely on carefully designed fusion strategies that either use modality-specific adaptations or rely on loosely coupled interactions, thereby limiting flexibility and resulting in less effective cross-modal coordination. Moreover, these methods often struggle to balance efficient information exchange with preserving the unique characteristics of each modality across different modality combinations. To address these challenges, we propose CrossWeaver, a simple yet effective multimodal fusion framework for arbitrary-modality semantic segmentation. Its core is a Modality Interaction Block (MIB), which enables selective and reliability-aware cross-modal interaction within the encoder, while a lightweight Seam-Aligned Fusion (SAF) module further aggregates the enhanced features. Extensive experiments on multiple multimodal semantic segmentation benchmarks demonstrate that our framework achieves state-of-the-art performance with minimal additional parameters and strong generalization to unseen modality combinations.