mlsanityData to 14 Oct 2026

Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation

arXiv 2610.17800 ↗First posted 13 Oct 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-modal semantic segmentation adds an auxiliary sensing stream to RGB images (RGB-X), such as thermal, LiDAR, depth, event, polarization, or light field. This can improve robustness in poor lighting and under motion blur. But different sensors often produce features that do not match well, and fusion can become unstable. To alleviate these issues, we propose a bidirectional polarity-aware cross-modality fusion (BPCF) module that effectively captures complementary cues while enhancing feature alignment. We evaluate the framework on six modality pairings: RGB-Thermal, RGB-LiDAR, RGB-Depth, RGB-Event, RGB-Polarization, and RGB-Light Field. Our model achieve state-of-the-art (SOTA) results on eight public datasets, including MFNet, KITTI-360, DELIVER, DDD17, DSEC, MCubeS, ZJU, and UrbanLF. The codes and trained models will be released upon acceptance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 57 of its 76 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

57 of 76 references.

References · 57

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  6. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  7. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  8. 2016Pyramid Scene Parsing Networkcited by 2,479
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  11. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  12. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
38 more
  1. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
  2. 2021DSEC: A Stereo Event Camera Dataset for Driving Scenarioscited by 303
  3. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  4. 2019Events-to-Video: Bringing Modern Computer Vision to Event Camerascited by 193
  5. 2019Video to Events: Recycling Video Datasets for Event Camerascited by 165
  6. 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 142
  7. 2022Multimodal Token Fusion for Vision Transformerscited by 136
  8. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 129
  9. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
  10. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
  11. 2021Event-Based Fusion for Motion Deblurring with Cross-modal Attentioncited by 113
  12. 2019In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Imagescited by 100
  13. 2018EV-SegNet: Semantic Segmentation for Event-based Camerascited by 93
  14. 2017DDD17: End-To-End DAVIS Driving Datasetcited by 84
  15. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 84
  16. 2022ESS: Learning Event-based Semantic Segmentation from Still Imagescited by 82
  17. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 60
  18. 2021Decoupled Dynamic Filter Networkscited by 53
  19. 2020Dynamic Region-Aware Convolutioncited by 50
  20. 2021Temporal Memory Attention for Video Semantic Segmentationcited by 50
  21. 2021EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillationcited by 47
  22. 2023Hierarchical Neural Memory Network for Low Latency Event Processingcited by 47
  23. 2021Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsingcited by 45
  24. 2024GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformercited by 44
  25. 2022RGB-Event Fusion for Moving Object Detection in Autonomous Drivingcited by 40
  26. 2024OpenESS: Event-based Semantic Scene Understanding with Open Vocabulariescited by 38
  27. 2021Bridging the Gap between Events and Frames through Unsupervised Domain Adaptationcited by 37
  28. 2021Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translationcited by 32
  29. 2025PolaFormer: Polarity-aware Linear Attention for Vision Transformerscited by 32
  30. 2022HRFuser: A Multi-resolution Sensor Fusion Architecture for 2D Object Detectioncited by 31
  31. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 31
  32. 2024Embracing Events and Frames with Hierarchical Feature Refinement Network for Object Detectioncited by 28
  33. 2022HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalitiescited by 25
  34. 2020Polarization-driven Semantic Segmentation via Efficient Attention-bridged Fusioncited by 24
  35. 2021Exploring Event-driven Dynamic Context for Accident Scene Segmentationcited by 23
  36. 2024Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentationcited by 22
  37. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 14
  38. 2022DooDLeNet: Double DeepLab Enhanced Feature Fusion for Thermal-color Semantic Segmentationcited by 13

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.