mlsanityData to 7 Oct 2026

Long Movie Clip Classification with State-Space Video Models

arXiv 2204.01692 ↗First posted 4 Apr 2022cs.CVcited by 106

Not on the map. Why?

Cited by 34 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most modern video recognition models are designed to operate on short video clips (e.g., 5-10s in length). Thus, it is challenging to apply such models to long movie understanding tasks, which typically require sophisticated long-range temporal reasoning. The recently introduced video transformers partially address this issue by using long-range temporal self-attention. However, due to the quadratic cost of self-attention, such models are often costly and impractical to use. Instead, we propose ViS4mer, an efficient long-range video model that combines the strengths of self-attention and the recently introduced structured state-space sequence (S4) layer. Our model uses a standard Transformer encoder for short-range spatiotemporal feature extraction, and a multi-scale temporal S4 decoder for subsequent long-range temporal reasoning. By progressively reducing the spatiotemporal feature resolution and channel dimension at each decoder layer, ViS4mer learns complex long-range spatiotemporal dependencies in a video. Furthermore, ViS4mer is 2.63× faster and requires 8× less GPU memory than the corresponding pure self-attention-based model. Additionally, ViS4mer achieves state-of-the-art results in 6 out of 9 long-form movie video classification tasks on the Long Video Understanding (LVU) benchmark. Furthermore, we show that our approach successfully generalizes to other domains, achieving competitive results on the Breakfast and the COIN procedural activity datasets. The code is publicly available at: https://github.com/md-mohaiminul/ViS4mer.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 55 of its 61 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

55 of 61 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2017Mask R-CNNcited by 7,259
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2021ViViT: A Video Vision Transformercited by 1,610
  13. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  14. 2020Rethinking Attention with Performerscited by 1,584
  15. 2021Video Swin Transformercited by 1,090
  16. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  17. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  18. 2021Multiscale Vision Transformerscited by 896
  19. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  20. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  21. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  22. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  23. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  24. 2018A Short Note about Kinetics-600cited by 331
  25. 2017Temporal Relational Reasoning in Videoscited by 324
  26. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  27. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  28. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  29. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  30. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  31. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  32. 2021Towards Long-Form Video Understandingcited by 146
  33. 2018ECO: Efficient Convolutional Network for Online Video Understandingcited by 115
  34. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  35. 2018Timeception for Complex Action Recognitioncited by 89
  36. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  37. 2017MovieGraphs: Towards Understanding Human-Centric Situations from Videoscited by 67
  38. 2019VideoGraph: Recognizing Minutes-Long Human Activities in Videoscited by 57

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 17; 2024: 44; 2025: 28; 2026: 10.

106 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  3. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  5. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  6. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  7. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  8. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  9. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  10. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  11. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  12. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Datacited by 84
38 more
  1. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  2. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  3. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  4. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  5. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  6. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  7. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  8. 2024A Survey on Visual Mambacited by 50
  9. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  10. 2022Temporally Consistent Transformers for Video Generationcited by 43
  11. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41
  12. 2024U-shaped Vision Mamba for Single Image Dehazingcited by 40
  13. 2024ReMamber: Referring Image Segmentation with Mamba Twistercited by 38
  14. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  15. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  16. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  17. 2022CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Groundingcited by 27
  18. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  19. 2024VideoMamba: Spatio-Temporal Selective State Space Modelcited by 25
  20. 2024DVMSR: Distillated Vision Mamba for Efficient Super-Resolutioncited by 23
  21. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  22. 2024Video Token Merging for Long-form Video Understandingcited by 22
  23. 2024Mamba-FETrack: Frame-Event Tracking via State Space Modelcited by 21
  24. 2024MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Promptcited by 20
  25. 2024MambaTrack: A Simple Baseline for Multiple Object Tracking with State Space Modelcited by 18
  26. 2024Slot State Space Modelscited by 17
  27. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  28. 2025BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimationcited by 17
  29. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  30. 2024HSIMamba: Hyperpsectral Imaging Efficient Feature Learning with Bidirectional State Space for Classificationcited by 14
  31. 2024Enhancing Long Video Understanding via Hierarchical Event-Based Memorycited by 14
  32. 2024FD-Vision Mamba for Endoscopic Exposure Correctioncited by 13
  33. 2024Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithmscited by 11
  34. 2024HERMES: temporal-coHERent long-forM understanding with Episodes and Semanticscited by 11
  35. 2024Activating Wider Areas in Image Super-Resolutioncited by 10
  36. 2023ENTL: Embodied Navigation Trajectory Learnercited by 9
  37. 2024Encoding and Controlling Global Semantics for Long-form Video Question Answeringcited by 8
  38. 2023Long-range Multimodal Pretraining for Movie Understandingcited by 7

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  2. 2025Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders
  3. 2025ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding
  4. 2022Frozen CLIP Models are Efficient Video Learners
  5. 2025Flow4Agent: Long-form Video Understanding via Motion Prior from Optical Flow
  6. 2026Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding
  7. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding
  8. 2024Streaming Long Video Understanding with Large Language Models