VideoSEMA: a scalable and efficient Mamba-like attention for video understanding
On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models
Abstract (arXiv)
We present for video understanding (classification) a split space-time attention model, VideoSEMA, consisting of a scalable and efficient Mamba-like attention (SEMA) block in space and a softmax temporal attention in time. In each frame, SEMA attention applies a local window attention in parallel with a global averaging in a Mamba macro-architecture, which is called Mamba-like. Under certain rank conditions, we prove that the computationally cheaper split space-time attention is equivalent to full space-time attention. On benchmark K400 data sets, VideoSEMA out-performs heavier vision transformer and Mamba models. On benchmark SSv2 data, VideoSEMA leads in top-1 accuracy among models of similar parameter sizes. As image resolution scales up from standard 2242 to 10242 on K400 and without fine-tuning, VideoSEMA degrades much more gracefully than VideoMamba in accuracy. It is promising to extend VideoSEMA to longer videos with a dilated/sparse temporal attention.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 20 of its 25 references that are in the map. First posted on arXiv 16 Jul 2026.
Builds on
20 of 25 references.
References · 20
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
- 2021Video Swin Transformercited by 1,091
- 2021Multiscale Vision Transformerscited by 897
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
- 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
- 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
- 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
- 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
8 more
- 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
- 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
- 2020TDN: Temporal Difference Networks for Efficient Action Recognitioncited by 183
- 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 102
- 2021CT-Net: Channel Tensorization Network for Video Classificationcited by 26
- 2025Make Your Training Flexible: Towards Deployment-Efficient Video Modelscited by 6
- 2024A Primal-Dual Framework for Transformers and Neural Networkscited by 4
- 2024AFIDAF: Alternating Fourier and Image Domain Adaptive Filters as an Efficient Alternative to Attention in ViTscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Snakes and Ladders: Two Steps Up for VideoMamba
- 2025SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging
- 2024VideoMamba: Spatio-Temporal Selective State Space Model
- 2024VideoMamba: State Space Model for Efficient Video Understanding
- 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
- 2025A Separable Self-attention Inspired by the State Space Model for Computer Vision
- 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding
- 2025PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation