mlsanityData to 7 Oct 2026

Selective State Space Memory for Large Vision-Language Models

arXiv 2412.09875 ↗First posted 13 Dec 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across a wide range of multimodal tasks. However, fine-tuning these models for domain-specific applications remains a computationally intensive challenge. This paper introduces State Space Memory Integration (SSMI), a novel approach for efficient fine-tuning of LVLMs. By integrating lightweight Mamba-based state space modules into the LVLM architecture, SSMI captures long-range dependencies and injects task-specific visual and sequential patterns effectively. Unlike traditional fine-tuning methods, SSMI requires only a fraction of the model's parameters to be updated, making it computationally efficient and scalable. Experiments on benchmark datasets, including COCO Captioning, VQA, and Flickr30k, demonstrate that SSMI achieves state-of-the-art performance while maintaining robustness and generalization capabilities. Comprehensive analysis further validates the advantages of SSMI in terms of efficiency, adaptability, and interpretability, positioning it as a compelling solution for fine-tuning large-scale vision-language models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 22 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

12 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models25 Oct
  2. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens7 Oct

Other references · 10

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  4. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  5. 2024Visual Mamba: A Survey and New Outlookscited by 41
  6. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  7. 2024MemoryMamba: Memory-Augmented State Space Model for Defect Recognitioncited by 24
  8. 2024InsectMamba: Insect Pest Classification with State Space Modelcited by 11
  9. 2024A Mamba Foundation Model for Time Series Forecastingcited by 9
  10. 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  2. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
  3. 2026Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders
  4. 2024VL-Mamba: Exploring State Space Models for Multimodal Learning
  5. 2025Towards General Continuous Memory for Vision-Language Models
  6. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  7. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
  8. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges