# Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers

arXiv: https://arxiv.org/abs/2503.11579 · HTML: https://mlsanity.com/paper/2503.11579/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 75 of its 81 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)
- State-space sequence models (S4, Mamba) / [Video, motion and generative Mamba models](https://mlsanity.com/ssm/video-motion-and-generative-mamba-models/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 75)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)

## Built on by (most cited first, 30 of 36)

- [TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models](https://mlsanity.com/paper/2512.02014/) (2025)
- [VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation](https://mlsanity.com/paper/2505.14640/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation](https://mlsanity.com/paper/2604.24763/) (2026)
- [REVISOR: Beyond Textual Reflection, Towards Multimodal Introspective Reasoning in Long-Form Video Understanding](https://mlsanity.com/paper/2511.13026/) (2025)
- [AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding](https://mlsanity.com/paper/2506.13589/) (2025)
- [ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers](https://mlsanity.com/paper/2504.00502/) (2025)
- [SiLVR: A Simple Language-based Video Reasoning Framework](https://mlsanity.com/paper/2505.24869/) (2025)
- [ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding](https://mlsanity.com/paper/2602.16412/) (2026)
- [TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2503.24067/) (2025)
- [IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2504.15415/) (2025)
- [Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoning](https://mlsanity.com/paper/2510.15440/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding](https://mlsanity.com/paper/2511.16595/) (2025)
- [Think with Grounding: Curriculum Reinforced Reasoning with Video Grounding for Long Video Understanding](https://mlsanity.com/paper/2602.18702/) (2026)
- [Native Active Perception as Reasoning for Omni-Modal Understanding](https://mlsanity.com/paper/2606.19341/) (2026)
- [State-Space Hierarchical Compression with Gated Attention and Learnable Sampling for Hour-Long Video Understanding in Large Multimodal Models](https://mlsanity.com/paper/2506.13564/) (2025)
- [Enhancing Long Video Question Answering with Scene-Localized Frame Grouping](https://mlsanity.com/paper/2508.03009/) (2025)
- [MixerCSeg: An Efficient Mixer Architecture for Crack Segmentation via Decoupled Mamba Attention](https://mlsanity.com/paper/2603.01361/) (2026)
- [On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks](https://mlsanity.com/paper/2504.13209/) (2025)
- [Lightweight Quad Bayer HybridEVS Demosaicing via State Space Augmented Cross-Attention](https://mlsanity.com/paper/2508.06058/) (2025)
- [Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy](https://mlsanity.com/paper/2509.17901/) (2025)
- [Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding](https://mlsanity.com/paper/2512.00805/) (2025)
- [Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoning](https://mlsanity.com/paper/2604.04372/) (2026)
- [VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding](https://mlsanity.com/paper/2606.05259/) (2026)
- [Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding](https://mlsanity.com/paper/2607.28516/) (2026)
- [HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Pooling](https://mlsanity.com/paper/2510.23043/) (2025)
- [MambaVF: State Space Model for Efficient Video Fusion](https://mlsanity.com/paper/2602.06017/) (2026)
- [Linear Scaling Video VLMs for Long Video Understanding](https://mlsanity.com/paper/2605.31598/) (2026)
- [Evaluating Multimodal Narrative Understanding of Popular Hollywood Films](https://mlsanity.com/paper/2608.21430/) (2026)

## Abstract (arXiv)

State-of-the-art transformer-based large multimodal models (LMMs) struggle to handle hour-long video inputs due to the quadratic complexity of the causal self-attention operations, leading to high computational costs during training and inference. Existing token compression-based methods reduce the number of video tokens but often incur information loss and remain inefficient for extremely long sequences. In this paper, we explore an orthogonal direction to build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to encode video tokens with linear complexity. Without any token reduction, VAMBA can encode more than 1024 frames (640×360) on a single GPU, while transformer-based models can only encode 256 frames. On long video input, VAMBA achieves at least 50% reduction in GPU memory usage during training and inference, and nearly doubles the speed per training step compared to transformer-based LMMs. Our experimental results demonstrate that VAMBA improves accuracy by 4.3% on the challenging hour-long video understanding benchmark LVBench over prior efficient video LMMs, and maintains strong performance on a broad spectrum of long and short video understanding tasks.