mlsanityData to 7 Oct 2026

AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment

arXiv 2412.00833 ↗First posted 1 Dec 2024cs.CV cs.AIcited by 17

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

Abstract (arXiv)

Cross-modal alignment is crucial for multimodal representation fusion due to the inherent heterogeneity between modalities. While Transformer-based methods have shown promising results in modeling inter-modal relationships, their quadratic computational complexity limits their applicability to long-sequence or large-scale data. Although recent Mamba-based approaches achieve linear complexity, their sequential scanning mechanism poses fundamental challenges in comprehensively modeling cross-modal relationships. To address this limitation, we propose AlignMamba, an efficient and effective method for multimodal fusion. Specifically, grounded in Optimal Transport, we introduce a local cross-modal alignment module that explicitly learns token-level correspondences between different modalities. Moreover, we propose a global cross-modal alignment loss based on Maximum Mean Discrepancy to implicitly enforce the consistency between different modal distributions. Finally, the unimodal representations after local and global alignment are passed to the Mamba backbone for further cross-modal interaction and multimodal fusion. Extensive experiments on complete and incomplete multimodal fusion tasks demonstrate the effectiveness and efficiency of the proposed method.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 33 of its 61 references that are in the map. First posted on arXiv 1 Dec 2024.

Builds on

33 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment8 Oct
  2. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders2 Jul
  3. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference21 Mar
  4. 2024VL-Mamba: Exploring State Space Models for Multimodal Learning20 Mar
  5. 2024Pan-Mamba: Effective pan-sharpening with State Space Model19 Feb

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  11. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  12. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
16 more
  1. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  2. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  3. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  4. 2016On Deep Multi-View Representation Learning: Objectives and Optimizationcited by 162
  5. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  6. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  7. 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 86
  8. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  9. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
  10. 2024Fusion-Mamba for Cross-modality Object Detectioncited by 67
  11. 2021Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysiscited by 41
  12. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  13. 2024MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusioncited by 35
  14. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  15. 2024Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalizationcited by 24
  16. 2024SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025Signal: Selective Interaction and Global-local Alignment for Multi-Modal Object Re-Identificationcited by 4
  2. 2025Unlocking Multi-Modal Potentials for Link Prediction on Dynamic Text-Attributed Graphscited by 3
  3. 2025LIDAR: Lightweight Adaptive Cue-Aware Fusion Vision Mamba for Multimodal Segmentation of Structural Crackscited by 3
  4. 2025LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mambacited by 3
  5. 2025Lightweight Quad Bayer HybridEVS Demosaicing via State Space Augmented Cross-Attentioncited by 2
  6. 2025ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysiscited by 2
  7. 2025ARPGNet: Appearance- and Relation-aware Parallel Graph Attention Fusion Network for Facial Expression Recognitioncited by 1
  8. 2026Walking Further: Semantic-aware Multimodal Gait Recognition Under Long-Range Conditionscited by 1
  9. 2026Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysiscited by 1
  10. 2025Vision-Proprioception Fusion with Mamba2 in End-to-End Reinforcement Learning for Motion Controlcited by 0
  11. 2025CPCLDETECTOR: Knowledge Enhancement and Alignment Selection for Chinese Patronizing and Condescending Language Detectioncited by 0
  12. 2025MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulationcited by 0
5 more
  1. 2026AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mambacited by 0
  2. 2026MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Modelscited by 0
  3. 2026Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMscited by 0
  4. 2026CALM: Interpretable Cross-Modal Alignment for Biomarker Discovery from Unpaired Datacited by 0
  5. 2026CASS: Contribution-Aware Structured Sparsity for Model Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Alignment and Fusion: A Survey
  2. 2026CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook
  3. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  4. 2026LatentUMM: Dual Latent Alignment for Unified Multimodal Models
  5. 2026Anisotropic Modality Align
  6. 2024ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map
  7. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion
  8. 2026Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models