# VideoMamba: Spatio-Temporal Selective State Space Model

arXiv: https://arxiv.org/abs/2407.08476 · HTML: https://mlsanity.com/paper/2407.08476/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 48 of its 54 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Video, motion and generative Mamba models](https://mlsanity.com/ssm/video-motion-and-generative-mamba-models/)

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)

## Built on by (most cited first, 25 of 25)

- [STORM: Token-Efficient Long Video Understanding for Multimodal LLMs](https://mlsanity.com/paper/2503.04130/) (2025)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding](https://mlsanity.com/paper/2507.02591/) (2025)
- [Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generation](https://mlsanity.com/paper/2411.01647/) (2024)
- [Moment Sampling in Video LLMs for Long-Form Video QA](https://mlsanity.com/paper/2507.00033/) (2025)
- [Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation](https://mlsanity.com/paper/2411.15224/) (2024)
- [High-Resolution Spatiotemporal Modeling with Global-Local State Space Models for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2510.11017/) (2025)
- [MVQA: Mamba with Unified Sampling for Efficient Video Quality Assessment](https://mlsanity.com/paper/2504.16003/) (2025)
- [InterMamba: Efficient Human-Human Interaction Generation with Adaptive Spatio-Temporal Mamba](https://mlsanity.com/paper/2506.03084/) (2025)
- [PASS: Path-selective State Space Model for Event-based Recognition](https://mlsanity.com/paper/2409.16953/) (2024)
- [Identifying Spatio-Temporal Drivers of Extreme Events](https://mlsanity.com/paper/2410.24075/) (2024)
- [Empathetic Response in Audio-Visual Conversations Using Emotion Preference Optimization and MambaCompressor](https://mlsanity.com/paper/2412.17572/) (2024)
- [BrainMT: A Hybrid Mamba-Transformer Architecture for Modeling Long-Range Dependencies in Functional MRI Data](https://mlsanity.com/paper/2506.22591/) (2025)
- [HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Pooling](https://mlsanity.com/paper/2510.23043/) (2025)
- [RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation](https://mlsanity.com/paper/2603.24295/) (2026)
- [HiCrew: Hierarchical Reasoning for Long-Form Video Understanding via Question-Aware Multi-Agent Collaboration](https://mlsanity.com/paper/2604.21444/) (2026)
- [ReynoldsFlow: Physics-Inspired Spatiotemporal Flow Representation for Video Understanding](https://mlsanity.com/paper/2503.04500/) (2025)
- [Mamba-3D as Masked Autoencoders for Accurate and Data-Efficient Analysis of Medical Ultrasound Videos](https://mlsanity.com/paper/2503.20258/) (2025)
- [Towards Efficient Vision State Space Models via Token Merging](https://mlsanity.com/paper/2508.13599/) (2025)
- [Time-Scaling State-Space Models for Dense Video Captioning](https://mlsanity.com/paper/2509.03426/) (2025)
- [Beyond Real versus Fake Towards Intent-Aware Video Analysis](https://mlsanity.com/paper/2511.22455/) (2025)
- [Geometric Transformation-Embedded Mamba for Learned Video Compression](https://mlsanity.com/paper/2603.07912/) (2026)
- [TrajMamba: An Ego-Motion-Guided Mamba Model for Pedestrian Trajectory Prediction from an Egocentric Perspective](https://mlsanity.com/paper/2603.14739/) (2026)
- [Stitch-Inferencer: Enhance Endoscopic Video Segmentation and Tracking via Panoramic Reconstruction](https://mlsanity.com/paper/2607.14968/) (2026)
- [Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs](https://mlsanity.com/paper/2609.10355/) (2026)

## Abstract (arXiv)

We introduce VideoMamba, a novel adaptation of the pure Mamba architecture, specifically designed for video recognition. Unlike transformers that rely on self-attention mechanisms leading to high computational costs by quadratic complexity, VideoMamba leverages Mamba's linear complexity and selective SSM mechanism for more efficient processing. The proposed Spatio-Temporal Forward and Backward SSM allows the model to effectively capture the complex relationship between non-sequential spatial and sequential temporal information in video. Consequently, VideoMamba is not only resource-efficient but also effective in capturing long-range dependency in videos, demonstrated by competitive performance and outstanding efficiency on a variety of video understanding benchmarks. Our work highlights the potential of VideoMamba as a powerful tool for video understanding, offering a simple yet effective baseline for future research in video analysis.