mlsanityData to 7 Oct 2026

VL-Mamba: Exploring State Space Models for Multimodal Learning

arXiv 2403.13600 ↗First posted 20 Mar 2024cs.CVcited by 93

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) have attracted widespread interest and have rich applications. However, the inherent attention mechanism in its Transformer structure requires quadratic complexity and results in expensive computational overhead. Therefore, in this work, we propose VL-Mamba, a multimodal large language model based on state space models, which have been shown to have great potential for long-sequence modeling with fast inference and linear scaling in sequence length. Specifically, we first replace the transformer-based backbone language model such as LLama or Vicuna with the pre-trained Mamba language model. Then, we empirically explore how to effectively apply the 2D vision selective scan mechanism for multimodal learning and the combinations of different vision encoders and variants of pretrained Mamba language models. The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 58 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

55 of 58 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  2. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones28 Dec
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
32 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  13. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  14. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  15. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  16. 2024VMamba: Visual State Space Modelcited by 955
  17. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  18. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  19. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  20. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  21. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  22. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  23. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  24. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  25. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  26. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  27. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  28. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  29. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  30. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  31. 2020Referring Expression Comprehension: A Survey of Methods and Datasetscited by 78
  32. 2024Vivim: a Video Vision Mamba for Medical Video Segmentationcited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 39; 2026: 13.

93 papers.

Sort:

By area:

  1. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  2. 2024MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detectioncited by 113
  3. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  4. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  5. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  6. 2024FreqMamba: Viewing Mamba from a Frequency Perspective for Image Derainingcited by 57
  7. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 53
  8. 2024A Survey on Visual Mambacited by 50
  9. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  10. 2024Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusioncited by 40
  11. 2024A Survey of Mambacited by 39
  12. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
38 more
  1. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  2. 2024MambaVC: Learned Visual Compression with Selective State Spacescited by 28
  3. 2024QMambaBSR: Burst Image Super-Resolution with Query State Space Modelcited by 27
  4. 2025Directing Mamba to Complex Textures: An Efficient Texture-Aware State Space Model for Image Restorationcited by 27
  5. 2024NetMamba: Efficient Network Traffic Classification via Pre-training Unidirectional Mambacited by 24
  6. 2024DVMSR: Distillated Vision Mamba for Efficient Super-Resolutioncited by 23
  7. 2024MVGamba: Unify 3D Content Generation as State Space Sequence Modelingcited by 23
  8. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  9. 2025Linear Attention Modeling for Learned Image Compressioncited by 22
  10. 2024The Evolution of Multimodal Model Architecturescited by 21
  11. 2024Hybrid Mamba for Few-Shot Segmentationcited by 21
  12. 2024Mamba in Vision: A Comprehensive Survey of Techniques and Applicationscited by 18
  13. 2024AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignmentcited by 17
  14. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  15. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  16. 2025AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentationcited by 15
  17. 2024S3Mamba: Arbitrary-Scale Super-Resolution via Scaleable State Space Modelcited by 12
  18. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  19. 2024GOMAA-Geo: GOal Modality Agnostic Active Geo-localizationcited by 11
  20. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillationcited by 11
  21. 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Modelscited by 11
  22. 2024VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Modelscited by 10
  23. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  24. 2024CoMamba: Real-time Cooperative Perception Unlocked with State Space Modelscited by 9
  25. 2024Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Surveycited by 9
  26. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  27. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  28. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  29. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  30. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  31. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 7
  32. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevancecited by 7
  33. 2024Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modelingcited by 6
  34. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
  35. 2025MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modelingcited by 6
  36. 2025Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environmentscited by 6
  37. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  38. 2024Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Datacited by 5

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2
  2. 2024Multimodal Instruction Tuning with Hybrid State Space Models
  3. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference
  4. 2024LocalMamba: Visual State Space Model with Windowed Selective Scan
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2024Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model
  7. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  8. 2025M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer