# VideoMamba: State Space Model for Efficient Video Understanding

arXiv: https://arxiv.org/abs/2403.06977 · HTML: https://mlsanity.com/paper/2403.06977/

## Key facts

- Cited by 328 papers in the map (data through 14 Oct 2026).
- Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 76 of its 95 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Video, motion and generative Mamba models](https://mlsanity.com/ssm/video-motion-and-generative-mamba-models/)

## Builds on (most cited first, 30 of 76)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)

## Built on by (most cited first, 30 of 328)

- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [Is Mamba Effective for Time Series Forecasting?](https://mlsanity.com/paper/2403.11144/) (2024)
- [UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation](https://mlsanity.com/paper/2406.01188/) (2024)
- [ZigMa: A DiT-style Zigzag Mamba Diffusion Model](https://mlsanity.com/paper/2403.13802/) (2024)
- [DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis](https://mlsanity.com/paper/2405.14224/) (2024)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model](https://mlsanity.com/paper/2405.14174/) (2024)
- [MEM: Multi-Scale Embodied Memory for Vision Language Action Models](https://mlsanity.com/paper/2603.03596/) (2026)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking](https://mlsanity.com/paper/2412.15691/) (2024)
- [RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining](https://mlsanity.com/paper/2407.21773/) (2024)
- [Vision Mamba: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2405.04404/) (2024)
- [Diffusion Models and Representation Learning: A Survey](https://mlsanity.com/paper/2407.00783/) (2024)
- [Vision-LSTM: xLSTM as Generic Vision Backbone](https://mlsanity.com/paper/2406.04303/) (2024)
- [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024)
- [STORM: Token-Efficient Long Video Understanding for Multimodal LLMs](https://mlsanity.com/paper/2503.04130/) (2025)
- [MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space](https://mlsanity.com/paper/2405.16105/) (2024)
- [Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba](https://mlsanity.com/paper/2407.09646/) (2024)
- [Visual Mamba: A Survey and New Outlooks](https://mlsanity.com/paper/2404.18861/) (2024)
- [Mamba-R: Vision Mamba ALSO Needs Registers](https://mlsanity.com/paper/2405.14858/) (2024)
- [Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion](https://mlsanity.com/paper/2410.15091/) (2024)
- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model](https://mlsanity.com/paper/2405.11831/) (2024)
- [TAPNext: Tracking Any Point (TAP) as Next Token Prediction](https://mlsanity.com/paper/2504.05579/) (2025)
- [Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers](https://mlsanity.com/paper/2503.11579/) (2025)
- [VSSD: Vision Mamba with Non-Causal State Space Duality](https://mlsanity.com/paper/2407.18559/) (2024)
- [Quamba: A Post-Training Quantization Recipe for Selective State Space Models](https://mlsanity.com/paper/2410.13229/) (2024)
- [Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More](https://mlsanity.com/paper/2502.03738/) (2025)
- [SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding](https://mlsanity.com/paper/2404.01174/) (2024)
- [Matten: Video Generation with Mamba-Attention](https://mlsanity.com/paper/2405.03025/) (2024)

## Abstract (arXiv)

Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain. The proposed VideoMamba overcomes the limitations of existing 3D convolution neural networks and video transformers. Its linear-complexity operator enables efficient long-term modeling, which is crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability in the visual domain without extensive dataset pretraining, thanks to a novel self-distillation technique; (2) Sensitivity for recognizing short-term actions even with fine-grained motion differences; (3) Superiority in long-term video understanding, showcasing significant advancements over traditional feature-based models; and (4) Compatibility with other modalities, demonstrating robustness in multi-modal contexts. Through these distinct advantages, VideoMamba sets a new benchmark for video understanding, offering a scalable and efficient solution for comprehensive video understanding. All the code and models are available at https://github.com/OpenGVLab/VideoMamba.