mlsanityData to 7 Oct 2026

Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM

arXiv 2606.04719 ↗First posted 3 Jun 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

The Transformer's quadratic complexity with input length imposes an unsustainable computational load on large language models (LLMs). In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. This paper explores a query-based cross-modal projector designed to bolster Mamba's efficiency for vision-language modeling by compressing visual tokens based on input through the cross-attention mechanism. This innovative projector also removes the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLM. Experimental results across various vision-language understanding benchmarks show that the proposed cross-modal projector enhances Mamba-based multimodal LLMs, boosting both performance and throughput.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 42 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

24 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference21 Mar
  2. 2024VL-Mamba: Exploring State Space Models for Multimodal Learning20 Mar
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
9 more
  1. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  2. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  4. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  5. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  7. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  8. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  9. 2024Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separationcited by 45

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2
  2. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation
  3. 2024CROME: Cross-Modal Adapters for Efficient Multimodal LLM
  4. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM
  5. 2025Small Vision-Language Models: A Survey on Compact Architectures and Techniques
  6. 2026Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
  7. 2025Rethinking Visual Information Processing in Multimodal LLMs
  8. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM