mlsanityData to 14 Oct 2026

Selective Structured State-Spaces for Long-Form Video Understanding

arXiv 2303.14526 ↗First posted 25 Mar 2023cs.CVcited by 105

On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models

also in 1 more
Abstract (arXiv)

Effective modeling of complex spatiotemporal dependencies in long-form videos remains an open problem. The recently proposed Structured State-Space Sequence (S4) model with its linear complexity offers a promising direction in this space. However, we demonstrate that treating all image-tokens equally as done by S4 model can adversely affect its efficiency and accuracy. To address this limitation, we present a novel Selective S4 (i.e., S5) model that employs a lightweight mask generator to adaptively select informative image tokens resulting in more efficient and accurate modeling of long-term spatiotemporal dependencies in videos. Unlike previous mask-based token reduction methods used in transformers, our S5 model avoids the dense self-attention calculation by making use of the guidance of the momentum-updated S4 model. This enables our model to efficiently discard less informative tokens and adapt to various long-form video understanding tasks more effectively. However, as is the case for most token reduction methods, the informative image tokens could be dropped incorrectly. To improve the robustness and the temporal horizon of our model, we propose a novel long-short masked contrastive learning (LSMCL) approach that enables our model to predict longer temporal context using shorter input videos. We present extensive comparative results using three challenging long-form video understanding datasets (LVU, COIN and Breakfast), demonstrating that our approach consistently outperforms the previous state-of-the-art S4 model by up to 9.6% accuracy while reducing its memory footprint by 23%.

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 66 of its 80 references that are in the map. First posted on arXiv 25 Mar 2023.

Builds on

66 of 80 references.

References · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2016Layer Normalizationcited by 5,669
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2017The Kinetics Human Action Video Datasetcited by 1,909
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2020Reformer: The Efficient Transformercited by 1,668
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  10. 2021ViViT: A Video Vision Transformercited by 1,620
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  12. 2020Rethinking Attention with Performerscited by 1,601
  13. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  14. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  15. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  16. 2021Video Swin Transformercited by 1,091
  17. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  18. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  19. 2021Multiscale Vision Transformerscited by 897
  20. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  21. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  22. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  23. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  24. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  25. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  26. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  27. 2018A Short Note about Kinetics-600cited by 333
  28. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  29. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  30. 2021Video Transformer Networkcited by 272
  31. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  32. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  33. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  34. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  35. 2021Towards Long-Form Video Understandingcited by 146
  36. 2018AdaFrame: Adaptive Frame Selection for Fast Video Recognitioncited by 134
  37. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 134
  38. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 128

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 61; 2025: 28; 2026: 12.

105 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  2. 2024VMamba: Visual State Space Modelcited by 969
  3. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 352
  4. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  5. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  6. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 167
  7. 2024Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentationcited by 133
  8. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  9. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 98
  10. 2024The Hidden Attention of Mamba Modelscited by 91
  11. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  12. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 65
38 more
  1. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 64
  2. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  3. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  4. 2024Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstructioncited by 56
  5. 2024Is Mamba Capable of In-Context Learning?cited by 45
  6. 2025MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classificationcited by 44
  7. 2024ReMamber: Referring Image Segmentation with Mamba Twistercited by 38
  8. 2024MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusioncited by 35
  9. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 33
  10. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 32
  11. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  12. 2024Weak-Mamba-UNet: Visual Mamba Makes CNN and ViT Work Better for Scribble-based Medical Image Segmentationcited by 29
  13. 2024MambaVC: Learned Visual Compression with Selective State Spacescited by 28
  14. 2024Exploring Token Pruning in Vision State Space Modelscited by 28
  15. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  16. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  17. 2024VideoMamba: Spatio-Temporal Selective State Space Modelcited by 25
  18. 2024DVMSR: Distillated Vision Mamba for Efficient Super-Resolutioncited by 24
  19. 2024Video Token Merging for Long-form Video Understandingcited by 23
  20. 2024Mamba-FETrack: Frame-Event Tracking via State Space Modelcited by 22
  21. 2024Semi-Mamba-UNet: Pixel-Level Contrastive and Pixel-Level Cross-Supervised Visual Mamba-based UNet for Semi-Supervised Medical Image Segmentationcited by 20
  22. 2024DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensingcited by 20
  23. 2024Slot State Space Modelscited by 17
  24. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  25. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  26. 2024HSIMamba: Hyperpsectral Imaging Efficient Feature Learning with Bidirectional State Space for Classificationcited by 14
  27. 2024Enhancing Long Video Understanding via Hierarchical Event-Based Memorycited by 14
  28. 2024What Can We Learn from State Space Models for Machine Learning on Graphs?cited by 13
  29. 2024A Mamba-based Siamese Network for Remote Sensing Change Detectioncited by 13
  30. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  31. 2024LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modelingcited by 12
  32. 2024HERMES: temporal-coHERent long-forM understanding with Episodes and Semanticscited by 11
  33. 2024Deep Continuous-Time State-Space Models for Marked Event Sequencescited by 11
  34. 2024Integrating Mamba Sequence Model and Hierarchical Upsampling Network for Accurate Semantic Segmentation of Multiple Sclerosis Legioncited by 8
  35. 2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusioncited by 8
  36. 2024Building Vision Models upon Heat Conductioncited by 7
  37. 2024MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imagingcited by 7
  38. 2024AdaCM2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reductioncited by 7

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026READ: Reading the Selective Scan for Faithful Video Attribution
  2. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding
  3. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
  4. 2026Video Understanding: Through A Temporal Lens
  5. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding
  6. 2026VideoTok4D: A 4D-Aware Video Tokenizer for Compact World Representation
  7. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
  8. 2024Motion Guided Token Compression for Efficient Masked Video Modeling