mlsanityData to 7 Oct 2026

Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference

arXiv 2403.14520 ↗First posted 21 Mar 2024cs.CVcited by 97

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

In recent years, the application of multimodal large language models (MLLM) in various fields has achieved remarkable success. However, as the foundation model for many downstream tasks, current MLLMs are composed of the well-known Transformer network, which has a less efficient quadratic computation complexity. To improve the efficiency of such basic models, we propose Cobra, a linear computational complexity MLLM. Specifically, Cobra integrates the efficient Mamba language model into the visual modality. Moreover, we explore and study various modal fusion schemes to create an effective multi-modal Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely competitive performance with current computationally efficient state-of-the-art methods, e.g., LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster speed due to Cobra's linear sequential modeling. (2) Interestingly, the results of closed-set challenging prediction benchmarks show that Cobra performs well in overcoming visual illusions and spatial relationship judgments. (3) Notably, Cobra even achieves comparable performance to LLaVA with about 43% of the number of parameters. We will make all codes of Cobra open-source and hope that the proposed method can facilitate future research on complexity problems in MLLM. Our project page is available at: https://sites.google.com/view/cobravlm.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 61 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

45 of 61 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
27 more
  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2019Root Mean Square Layer Normalizationcited by 1,425
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  11. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  13. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  15. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  16. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  17. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  18. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  19. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  20. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  21. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  22. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  23. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  24. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  25. 2024Stable LM 2 1.6B Technical Reportcited by 86
  26. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  27. 2022Instance-aware Prompt Learning for Language Understanding and Generationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 39; 2026: 15.

97 papers.

Sort:

By area:

  1. 2024MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detectioncited by 113
  2. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  3. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  4. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  5. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  6. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  7. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 53
  8. 2024A Survey on Visual Mambacited by 50
  9. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 48
  10. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  11. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  12. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
38 more
  1. 2024Visual Mamba: A Survey and New Outlookscited by 41
  2. 2024A Survey of Mambacited by 39
  3. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  4. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  5. 2025MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Modelscited by 27
  6. 2024CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Predictioncited by 26
  7. 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
  8. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
  9. 2024Matten: Video Generation with Mamba-Attentioncited by 24
  10. 2024MVGamba: Unify 3D Content Generation as State Space Sequence Modelingcited by 23
  11. 2025Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Frameworkcited by 23
  12. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  13. 2024The Evolution of Multimodal Model Architecturescited by 21
  14. 2024Hybrid Mamba for Few-Shot Segmentationcited by 21
  15. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstrationcited by 20
  16. 2024Mambular: A Sequential Model for Tabular Deep Learningcited by 18
  17. 2024Mamba in Vision: A Comprehensive Survey of Techniques and Applicationscited by 18
  18. 2024AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignmentcited by 17
  19. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  20. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  21. 2025AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentationcited by 15
  22. 2025Generative Models in Decision Making: A Surveycited by 14
  23. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 14
  24. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  25. 2024From Efficient Multimodal Models to World Models: A Surveycited by 13
  26. 2024Score and Distribution Matching Policy: Advanced Accelerated Visuomotor Policies via Matched Distillationcited by 12
  27. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  28. 2025RationalVLA: A Rational Vision-Language-Action Model with Dual Systemcited by 12
  29. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillationcited by 11
  30. 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Modelscited by 11
  31. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  32. 2024CoMamba: Real-time Cooperative Perception Unlocked with State Space Modelscited by 9
  33. 2024Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Surveycited by 9
  34. 2024QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learningcited by 9
  35. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  36. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  37. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  38. 2024Soft Masked Mamba Diffusion Model for CT to MRI Conversioncited by 7

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2
  2. 2024VL-Mamba: Exploring State Space Models for Multimodal Learning
  3. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance
  4. 2025M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer
  5. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2025Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation
  8. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies