mlsanityData to 7 Oct 2026

Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers

arXiv 2503.11579 ↗First posted 14 Mar 2025cs.CVcited by 35

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

also in 2 more
Abstract (arXiv)

State-of-the-art transformer-based large multimodal models (LMMs) struggle to handle hour-long video inputs due to the quadratic complexity of the causal self-attention operations, leading to high computational costs during training and inference. Existing token compression-based methods reduce the number of video tokens but often incur information loss and remain inefficient for extremely long sequences. In this paper, we explore an orthogonal direction to build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to encode video tokens with linear complexity. Without any token reduction, VAMBA can encode more than 1024 frames (640×360) on a single GPU, while transformer-based models can only encode 256 frames. On long video input, VAMBA achieves at least 50% reduction in GPU memory usage during training and inference, and nearly doubles the speed per training step compared to transformer-based LMMs. Our experimental results demonstrate that VAMBA improves accuracy by 4.3% on the challenging hour-long video understanding benchmark LVBench over prior efficient video LMMs, and maintains strong performance on a broad spectrum of long and short video understanding tasks.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 81 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

75 of 81 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMs6 Mar
  2. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling21 Jan
  3. 2024VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentation1 Dec
  4. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  5. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding22 Sep
  6. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  7. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  8. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos26 Aug
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models9 Aug
  10. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  11. 2024Long Context Transfer from Language to Vision24 Jun
  12. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
4 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 59

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  13. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  14. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  15. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  16. 2022Token Merging: Your ViT But Fastercited by 995
  17. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  18. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  19. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  20. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  21. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  22. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  23. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  24. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  25. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  26. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  27. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  28. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  29. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  30. 2024What matters when building vision-language models?cited by 321
  31. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  32. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  33. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  34. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  35. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  36. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  37. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  38. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 17.

35 papers.

Sort:

By area:

  1. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  2. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluationcited by 21
  3. 2026Large Language Model Reasoning Failurescited by 20
  4. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 17
  5. 2025REVISOR: Beyond Textual Reflection, Towards Multimodal Introspective Reasoning in Long-Form Video Understandingcited by 15
  6. 2025SiLVR: A Simple Language-based Video Reasoning Frameworkcited by 14
  7. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understandingcited by 14
  8. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  9. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Modelcited by 7
  10. 2026ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understandingcited by 7
  11. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMscited by 6
  12. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
23 more
  1. 2025Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoningcited by 4
  2. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  3. 2025State-Space Hierarchical Compression with Gated Attention and Learnable Sampling for Hour-Long Video Understanding in Large Multimodal Modelscited by 3
  4. 2025Enhancing Long Video Question Answering with Scene-Localized Frame Groupingcited by 3
  5. 2026Think with Grounding: Curriculum Reinforced Reasoning with Video Grounding for Long Video Understandingcited by 3
  6. 2026MixerCSeg: An Efficient Mixer Architecture for Crack Segmentation via Decoupled Mamba Attentioncited by 3
  7. 2026Native Active Perception as Reasoning for Omni-Modal Understandingcited by 3
  8. 2025On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attackscited by 2
  9. 2025Lightweight Quad Bayer HybridEVS Demosaicing via State Space Augmented Cross-Attentioncited by 2
  10. 2025Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understandingcited by 2
  11. 2026Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoningcited by 2
  12. 2026Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understandingcited by 2
  13. 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedycited by 1
  14. 2025HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Poolingcited by 1
  15. 2026MambaVF: State Space Model for Efficient Video Fusioncited by 1
  16. 2026Linear Scaling Video VLMs for Long Video Understandingcited by 1
  17. 2026VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understandingcited by 1
  18. 2026Evaluating Multimodal Narrative Understanding of Popular Hollywood Filmscited by 1
  19. 2025Gather-Scatter Mamba: Accelerating Propagation with Efficient State Space Modelcited by 0
  20. 2026LinMU: Multimodal Understanding Made Linearcited by 0
  21. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  22. 2026Zamba2-VL Technical Reportcited by 0
  23. 2026LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Stateful Token Reduction for Long-Video Hybrid VLMs
  2. 2024Snakes and Ladders: Two Steps Up for VideoMamba
  3. 2026APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention
  4. 2025Training-free Token Reduction for Vision Mamba
  5. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation
  6. 2025VideoMAP: Toward Scalable Mamba-based Video Autoregressive Pretraining
  7. 2026Geometric Transformation-Embedded Mamba for Learned Video Compression
  8. 2024Look Every Frame All at Once: Video-Ma2mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing