mlsanityData to 14 Oct 2026

Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation

arXiv 2404.04256 ↗First posted 5 Apr 2024cs.CVcited by 71

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

Abstract (arXiv)

Multi-modal semantic segmentation significantly enhances AI agents' perception and scene understanding, especially under adverse conditions like low-light or overexposed environments. Leveraging additional modalities (X-modality) like thermal and depth alongside traditional RGB provides complementary information, enabling more robust and reliable prediction. In this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic segmentation utilizing the advanced Mamba. Unlike conventional methods that rely on CNNs, with their limited local receptive fields, or Vision Transformers (ViTs), which offer global receptive fields at the cost of quadratic complexity, our model achieves global receptive fields with linear complexity. By employing a Siamese encoder and innovating a Mamba-based fusion mechanism, we effectively select essential information from different modalities. A decoder is then developed to enhance the channel-wise modeling ability of the model. Our proposed method is rigorously evaluated on both RGB-Thermal and RGB-Depth semantic segmentation tasks, demonstrating its superiority and marking the first successful application of State Space Models (SSMs) in multi-modal perception tasks. Code is available at https://github.com/zifuwan/Sigma.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 49 of its 77 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

49 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model17 Jan

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  8. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  9. 2024VMamba: Visual State Space Modelcited by 969
  10. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  12. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
36 more
  1. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  2. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 419
  3. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 352
  4. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
  5. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 314
  6. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  7. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 287
  8. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  9. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  10. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  11. 2022Multimodal Token Fusion for Vision Transformerscited by 136
  12. 2023Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentationcited by 129
  13. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
  14. 2020Deep Multimodal Fusion by Channel Exchangingcited by 118
  15. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
  16. 2018Learning a Discriminative Feature Network for Semantic Segmentationcited by 114
  17. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 113
  18. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 107
  19. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  20. 2019PST900: RGB-Thermal Calibration, Dataset and Segmentation Networkcited by 85
  21. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 84
  22. 2021ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentationcited by 77
  23. 2016Full-Resolution Residual Networks for Semantic Segmentation in Street Scenescited by 71
  24. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 60
  25. 2020Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentationcited by 48
  26. 2021FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentationcited by 45
  27. 2023Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Taskscited by 41
  28. 2022Efficient Multi-Task RGB-D Scene Analysis for Indoor Environmentscited by 33
  29. 2024Weak-Mamba-UNet: Visual Mamba Makes CNN and ViT Work Better for Scribble-based Medical Image Segmentationcited by 29
  30. 2023PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentationcited by 26
  31. 2024Context-Aware Interaction Network for RGB-T Semantic Segmentationcited by 21
  32. 2024HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generationcited by 14
  33. 2023Pixel Difference Convolutional Network for RGB-D Semantic Segmentationcited by 13
  34. 2023TransY-Net:Learning Fully Transformer Networks for Change Detection of Remote Sensing Imagescited by 13
  35. 2023Understanding Dark Scenes by Contrasting Multi-Modal Observationscited by 4
  36. 20242023 Low-Power Computer Vision Challenge (LPCVC) Summarycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 30; 2026: 19.

71 papers.

Sort:

By area:

  1. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  2. 2024Exploiting Multimodal Spatial-temporal Patterns for Video Object Trackingcited by 57
  3. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 54
  4. 2024A Survey on Visual Mambacited by 50
  5. 2024Visual Mamba: A Survey and New Outlookscited by 41
  6. 2024A Survey of Mambacited by 39
  7. 2024Hybrid Mamba for Few-Shot Segmentationcited by 21
  8. 2024MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Promptcited by 20
  9. 2024Mamba in Vision: A Comprehensive Survey of Techniques and Applicationscited by 18
  10. 2024MambaCSR: Dual-Interleaved Scanning for Compressed Image Super-Resolution With SSMscited by 14
  11. 2024A Mamba-based Siamese Network for Remote Sensing Change Detectioncited by 13
  12. 2024MambaDS: Near-Surface Meteorological Field Downscaling with Topography Constrained Selective State Space Modelingcited by 12
38 more
  1. 2024Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Surveycited by 9
  2. 2024STNMamba: Mamba-based Spatial-Temporal Normality Learning for Video Anomaly Detectioncited by 9
  3. 2024OneBEV: Using One Panoramic Image for Bird's-Eye-View Semantic Mappingcited by 8
  4. 2024RemoteDet-Mamba: A Hybrid Mamba-CNN Network for Multi-modal Object Detection in Remote Sensing Imagescited by 7
  5. 2025F-ViTA: Foundation Model Guided Visible to Thermal Translationcited by 7
  6. 2025UIS-Mamba: Exploring Mamba for Underwater Instance Segmentation via Dynamic Tree Scan and Hidden State Weakencited by 7
  7. 2025XFMamba: Cross-Fusion Mamba for Multi-View Medical Image Classificationcited by 6
  8. 2025Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detectioncited by 6
  9. 2025Referring Change Detection in Remote Sensing Imagerycited by 6
  10. 2025SpecMamba: Accelerating Mamba Inference on FPGA with Speculative Decodingcited by 5
  11. 2025Spatial-Frequency Enhanced Mamba for Multi-Modal Image Fusioncited by 5
  12. 2025Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentationcited by 4
  13. 2025MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognitioncited by 4
  14. 2024Alignment-free Raw Video Demoireingcited by 3
  15. 2024GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learningcited by 3
  16. 2024FTMoMamba: Motion Generation with Frequency and Text State Space Modelscited by 3
  17. 2025Cross-Modal Interactive Perception Network with Mamba for Lung Tumor Segmentation in PET-CT Imagescited by 3
  18. 2025Beyond Appearance: Geometric Cues for Robust Video Instance Segmentationcited by 3
  19. 2025LIDAR: Lightweight Adaptive Cue-Aware Fusion Vision Mamba for Multimodal Segmentation of Structural Crackscited by 3
  20. 2025End-to-End Multi-Modal Diffusion Mambacited by 3
  21. 2025Context-Gated Cross-Modal Perception with Visual Mamba for PET-CT Lung Tumor Segmentationcited by 3
  22. 2026NeXt2Former-CD: Efficient Remote Sensing Change Detection with Modern Vision Architecturescited by 3
  23. 2025Spectral-Aware Global Fusion for RGB-Thermal Semantic Segmentationcited by 2
  24. 2025Cross-modal State Space Modeling for Real-time RGB-thermal Wild Scene Semantic Segmentationcited by 2
  25. 2025CHARM: Collaborative Harmonization across Arbitrary Modalities for Modality-agnostic Semantic Segmentationcited by 2
  26. 2025TUNI: Unifying Pre-training and Fine-tuning with Modality-Aware Mutual Learning and Rectification for RGB-T Semantic Segmentationcited by 2
  27. 2025LEAF-Mamba: Local Emphatic and Adaptive Fusion State Space Model for RGB-D Salient Object Detectioncited by 2
  28. 2025DiffPixelFormer: Differential Pixel-Aware Transformer for RGB-D Indoor Scene Segmentationcited by 2
  29. 2024HRSAM: Efficient Interactive Segmentation in High-Resolution Imagescited by 1
  30. 2025MASTER: Multimodal Segmentation with Text Promptscited by 1
  31. 2025ECMNet:Lightweight Semantic Segmentation with Efficient CNN-Mamba Networkcited by 1
  32. 2025Power Battery Detectioncited by 1
  33. 2025Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detectioncited by 1
  34. 2025ChessMamba: Structure-Aware Interleaving of State Spaces for Change Detection in Remote Sensing Imagescited by 1
  35. 2026M3SR: Multi-Scale Multi-Perceptual Mamba for Efficient Spectral Reconstructioncited by 1
  36. 2026GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depthcited by 1
  37. 2026RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentationcited by 1
  38. 2026GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perceptioncited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MambaSeg: Harnessing Mamba for Accurate and Efficient Image-Event Semantic Segmentation
  2. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders
  3. 2026Generalizing Geometry-Guided Mamba as a Plug-and-Play Context Module for CNN-based Semantic Segmentation
  4. 2024A Novel State Space Model with Local Enhancement and State Sharing for Image Fusion
  5. 2025M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer
  6. 2024PyramidMamba: Rethinking Pyramid Feature Fusion with Selective Space State Model for Semantic Segmentation of Remote Sensing Imagery
  7. 2024Rethinking Scanning Strategies with Vision Mamba in Semantic Segmentation of Remote Sensing Imagery: An Experimental Study
  8. 2025MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks