# AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment

arXiv: https://arxiv.org/abs/2412.00833 · HTML: https://mlsanity.com/paper/2412.00833/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 33 of its 61 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Vision Mamba backbones](https://mlsanity.com/ssm/vision-mamba-backbones/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [On Deep Multi-View Representation Learning: Objectives and Optimization](https://mlsanity.com/paper/1602.01024/) (2016)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference](https://mlsanity.com/paper/2403.14520/) (2024)
- [VL-Mamba: Exploring State Space Models for Multimodal Learning](https://mlsanity.com/paper/2403.13600/) (2024)
- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis](https://mlsanity.com/paper/1911.05544/) (2019)
- [Decoupled Multimodal Distilling for Emotion Recognition](https://mlsanity.com/paper/2303.13802/) (2023)
- [Pan-Mamba: Effective pan-sharpening with State Space Model](https://mlsanity.com/paper/2402.12192/) (2024)
- [Fusion-Mamba for Cross-modality Object Detection](https://mlsanity.com/paper/2404.09146/) (2024)
- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusion](https://mlsanity.com/paper/2404.08406/) (2024)
- [HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding](https://mlsanity.com/paper/2404.13400/) (2024)
- [Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalization](https://mlsanity.com/paper/2402.18447/) (2024)

## Built on by (most cited first, 17 of 17)

- [Signal: Selective Interaction and Global-local Alignment for Multi-Modal Object Re-Identification](https://mlsanity.com/paper/2511.17965/) (2025)
- [Unlocking Multi-Modal Potentials for Link Prediction on Dynamic Text-Attributed Graphs](https://mlsanity.com/paper/2502.19651/) (2025)
- [LIDAR: Lightweight Adaptive Cue-Aware Fusion Vision Mamba for Multimodal Segmentation of Structural Cracks](https://mlsanity.com/paper/2507.22477/) (2025)
- [LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba](https://mlsanity.com/paper/2508.11849/) (2025)
- [Lightweight Quad Bayer HybridEVS Demosaicing via State Space Augmented Cross-Attention](https://mlsanity.com/paper/2508.06058/) (2025)
- [ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis](https://mlsanity.com/paper/2509.16900/) (2025)
- [ARPGNet: Appearance- and Relation-aware Parallel Graph Attention Fusion Network for Facial Expression Recognition](https://mlsanity.com/paper/2511.22188/) (2025)
- [Walking Further: Semantic-aware Multimodal Gait Recognition Under Long-Range Conditions](https://mlsanity.com/paper/2603.14189/) (2026)
- [Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs](https://mlsanity.com/paper/2604.05522/) (2026)
- [Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2604.12518/) (2026)
- [Vision-Proprioception Fusion with Mamba2 in End-to-End Reinforcement Learning for Motion Control](https://mlsanity.com/paper/2509.07593/) (2025)
- [CPCLDETECTOR: Knowledge Enhancement and Alignment Selection for Chinese Patronizing and Condescending Language Detection](https://mlsanity.com/paper/2509.18562/) (2025)
- [MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation](https://mlsanity.com/paper/2510.07328/) (2025)
- [AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba](https://mlsanity.com/paper/2603.18462/) (2026)
- [MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models](https://mlsanity.com/paper/2603.20074/) (2026)
- [CALM: Interpretable Cross-Modal Alignment for Biomarker Discovery from Unpaired Data](https://mlsanity.com/paper/2607.01656/) (2026)
- [CASS: Contribution-Aware Structured Sparsity for Model Merging](https://mlsanity.com/paper/2609.34184/) (2026)

## Abstract (arXiv)

Cross-modal alignment is crucial for multimodal representation fusion due to the inherent heterogeneity between modalities. While Transformer-based methods have shown promising results in modeling inter-modal relationships, their quadratic computational complexity limits their applicability to long-sequence or large-scale data. Although recent Mamba-based approaches achieve linear complexity, their sequential scanning mechanism poses fundamental challenges in comprehensively modeling cross-modal relationships. To address this limitation, we propose AlignMamba, an efficient and effective method for multimodal fusion. Specifically, grounded in Optimal Transport, we introduce a local cross-modal alignment module that explicitly learns token-level correspondences between different modalities. Moreover, we propose a global cross-modal alignment loss based on Maximum Mean Discrepancy to implicitly enforce the consistency between different modal distributions. Finally, the unimodal representations after local and global alignment are passed to the Mamba backbone for further cross-modal interaction and multimodal fusion. Extensive experiments on complete and incomplete multimodal fusion tasks demonstrate the effectiveness and efficiency of the proposed method.