mlsanityData to 7 Oct 2026

Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding

arXiv 2403.09626 ↗First posted 14 Mar 2024cs.CVcited by 106

On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models

Abstract (arXiv)

Understanding videos is one of the fundamental directions in computer vision research, with extensive efforts dedicated to exploring various architectures such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state space model, e.g., Mamba, shows promising traits to extend its success in long sequence modeling to video modeling. To assess whether Mamba can be a viable alternative to Transformers in the video understanding domain, in this work, we conduct a comprehensive set of studies, probing different roles Mamba can play in modeling videos, while investigating diverse tasks where Mamba could exhibit superiority. We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks. Our extensive experiments reveal the strong potential of Mamba on both video-only and video-language tasks while showing promising efficiency-performance trade-offs. We hope this work could provide valuable data points and insights for future research on video understanding. Code is public: https://github.com/OpenGVLab/video-mamba-suite.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 91 of its 132 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

91 of 132 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Selective Structured State-Spaces for Long-Form Video Understanding25 Mar

Other references · 90

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
38 more
  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  13. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  14. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  15. 2017The Kinetics Human Action Video Datasetcited by 1,894
  16. 2018SlowFast Networks for Video Recognitioncited by 1,862
  17. 2021ViViT: A Video Vision Transformercited by 1,610
  18. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  19. 2020Rethinking Attention with Performerscited by 1,584
  20. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  21. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  22. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  23. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  24. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  25. 2024VMamba: Visual State Space Modelcited by 955
  26. 2021Multiscale Vision Transformerscited by 896
  27. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  28. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  29. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  30. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  31. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  32. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  33. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  34. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  35. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  36. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  37. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  38. 2020Rescaling Egocentric Visioncited by 529

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 47; 2025: 41; 2026: 18.

106 papers.

Sort:

By area:

  1. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  2. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  3. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  4. 2024Fusion-Mamba for Cross-modality Object Detectioncited by 67
  5. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  6. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 53
  7. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  8. 2024A Survey on Visual Mambacited by 50
  9. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  10. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  11. 2024Visual Mamba: A Survey and New Outlookscited by 41
  12. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
38 more
  1. 2024Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusioncited by 40
  2. 2024MambaVC: Learned Visual Compression with Selective State Spacescited by 28
  3. 2024Matten: Video Generation with Mamba-Attentioncited by 24
  4. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesiscited by 22
  5. 2025Linear Attention Modeling for Learned Image Compressioncited by 22
  6. 2024PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learningcited by 21
  7. 2024SSUMamba: Spatial-Spectral Selective State Space Model for Hyperspectral Image Denoisingcited by 20
  8. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  9. 2024Serp-Mamba: Advancing High-Resolution Retinal Vessel Segmentation with Selective State-Space Modelcited by 20
  10. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  11. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  12. 2024Mamba in Vision: A Comprehensive Survey of Techniques and Applicationscited by 18
  13. 2024UniMD: Towards Unifying Moment Retrieval and Temporal Action Detectioncited by 16
  14. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  15. 2024GroupMamba: Efficient Group-Based Visual State Space Modelcited by 15
  16. 2025AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentationcited by 15
  17. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
  18. 2024SUM: Saliency Unification through Mamba for Visual Attention Modelingcited by 12
  19. 2025Auto-Regressive Diffusion for Generating 3D Human-Object Interactionscited by 12
  20. 2024MAMBA4D: Efficient Long-Sequence Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Modelscited by 11
  21. 2025M4V: Multimodal Mamba for Efficient Text-to-Video Generationcited by 11
  22. 2024S2Mamba: A Spatial-spectral State Space Model for Hyperspectral Image Classificationcited by 10
  23. 2025OpenTAD: A Unified Framework and Comprehensive Study of Temporal Action Detectioncited by 10
  24. 2025Animate-X++: Universal Character Image Animation with Dynamic Backgroundscited by 10
  25. 2025XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glassescited by 9
  26. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  27. 2024QMamba: On First Exploration of Vision Mamba for Image Quality Assessmentcited by 8
  28. 2024Harnessing Temporal Causality for Advanced Temporal Action Detectioncited by 8
  29. 2025LD-DETR: Loop Decoder DEtection TRansformer for Video Moment Retrieval and Highlight Detectioncited by 8
  30. 2025A Survey on Mamba Architecture for Vision Applicationscited by 8
  31. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  32. 2024Discrete Diffusion Language Model for Efficient Text Summarizationcited by 7
  33. 2024MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrievalcited by 7
  34. 2025Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challengecited by 6
  35. 2026EgoAction: Egocentric Action Composition with Reliability-Aware Temporal Fusion for the EPIC-KITCHENS Action Detection Challenge at CVPR 2026cited by 6
  36. 2024Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-trainingcited by 5
  37. 2025Temporal Action Detection Model Compression by Progressive Block Dropcited by 5
  38. 2025VSRM: A Robust Mamba-Based Framework for Video Super-Resolutioncited by 5

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VideoMamba: State Space Model for Efficient Video Understanding
  2. 2024MambaOut: Do We Really Need Mamba for Vision?
  3. 2024Snakes and Ladders: Two Steps Up for VideoMamba
  4. 2024VideoMamba: Spatio-Temporal Selective State Space Model
  5. 2024LocalMamba: Visual State Space Model with Windowed Selective Scan
  6. 2025Mamba-Adaptor: State Space Model Adaptor for Visual Recognition
  7. 2025VideoMAP: Toward Scalable Mamba-based Video Autoregressive Pretraining
  8. 2025VCMamba: Bridging Convolutions with Multi-Directional Mamba for Efficient Visual Representation