mlsanityData to 7 Oct 2026

Efficient Movie Scene Detection using State-Space Transformers

arXiv 2212.14427 ↗First posted 29 Dec 2022cs.CVcited by 51

Not on the map. Why?

Cited by 26 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to distinguish between different movie scenes is critical for understanding the storyline of a movie. However, accurately detecting movie scenes is often challenging as it requires the ability to reason over very long movie segments. This is in contrast to most existing video recognition models, which are typically designed for short-range video analysis. This work proposes a State-Space Transformer model that can efficiently capture dependencies in long movie videos for accurate movie scene detection. Our model, dubbed TranS4mer, is built using a novel S4A building block, which combines the strengths of structured state-space sequence (S4) and self-attention (A) layers. Given a sequence of frames divided into movie shots (uninterrupted periods where the camera position does not change), the S4A block first applies self-attention to capture short-range intra-shot dependencies. Afterward, the state-space operation in the S4A block is used to aggregate long-range inter-shot cues. The final TranS4mer model, which can be trained end-to-end, is obtained by stacking the S4A blocks one after the other multiple times. Our proposed TranS4mer outperforms all prior methods in three movie scene detection datasets, including MovieNet, BBC, and OVSD, while also being 2× faster and requiring 3× less GPU memory than standard Transformer models. We will release our code and models.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 29 Dec 2022.

Builds on

34 of 54 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
22 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  8. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  9. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  10. 2022Long Range Language Modeling via Gated State Spacescited by 265
  11. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  12. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  13. 2022Transformer Quality in Linear Timecited by 189
  14. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  15. 2021Towards Long-Form Video Understandingcited by 146
  16. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  17. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  18. 2020A Local-to-Global Approach to Multi-modal Movie Scene Segmentationcited by 60
  19. 2021Shot Contrastive Self-Supervised Learning for Scene Boundary Detectioncited by 35
  20. 2015A Deep Siamese Network for Scene Detection in Broadcast Videoscited by 21
  21. 2020Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluationcited by 20
  22. 2022Boundary-aware Self-supervised Learning for Video Scene Segmentationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 26; 2025: 16; 2026: 5.

51 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  2. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
  3. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  4. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  5. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Datacited by 84
  6. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 53
  7. 2025MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classificationcited by 44
  8. 2024ReMamber: Referring Image Segmentation with Mamba Twistercited by 38
  9. 2024MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusioncited by 35
  10. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  11. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  12. 2024DVMSR: Distillated Vision Mamba for Efficient Super-Resolutioncited by 23
38 more
  1. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  2. 2024MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Promptcited by 20
  3. 20232-D SSM: A General Spatial Layer for Visual Transformerscited by 18
  4. 2024MambaTrack: A Simple Baseline for Multiple Object Tracking with State Space Modelcited by 18
  5. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  6. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  7. 2024Enhancing Long Video Understanding via Hierarchical Event-Based Memorycited by 14
  8. 2024A Mamba-based Siamese Network for Remote Sensing Change Detectioncited by 13
  9. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  10. 2024LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modelingcited by 12
  11. 2024HERMES: temporal-coHERent long-forM understanding with Episodes and Semanticscited by 11
  12. 2024Activating Wider Areas in Image Super-Resolutioncited by 10
  13. 2025Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMscited by 9
  14. 2024MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imagingcited by 7
  15. 2024TRecViT: A Recurrent Video Transformercited by 5
  16. 2025SaMam: Style-aware State Space Model for Arbitrary Image Style Transfercited by 5
  17. 2024Multi-layer Learnable Attention Mask for Multimodal Taskscited by 4
  18. 2025Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesiscited by 4
  19. 2026MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering Benchmarkcited by 4
  20. 2025Diffusion Transformer-to-Mamba Distillation for High-Resolution Image Generationcited by 3
  21. 2024Dual Hyperspectral Mamba for Efficient Spectral Compressive Imagingcited by 2
  22. 2024ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mambacited by 2
  23. 2025MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videoscited by 2
  24. 2025MambaFlow: A Mamba-Centric Architecture for End-to-End Optical Flow Estimationcited by 2
  25. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  26. 2025MVQA: Mamba with Unified Sampling for Efficient Video Quality Assessmentcited by 2
  27. 2025Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Modelscited by 2
  28. 2024"Previously on ..." From Recaps to Story Summarizationcited by 1
  29. 2024Preserving Full Degradation Details for Blind Image Super-Resolutioncited by 1
  30. 2024HASN: Hybrid Attention Separable Network for Efficient Image Super-resolutioncited by 1
  31. 2025Cross-Modal Dual-Causal Learning for Long-Term Action Recognitioncited by 1
  32. 2025HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Poolingcited by 1
  33. 2026DACESR: Degradation-Aware Conditional Embedding for Real-World Image Super-Resolutioncited by 1
  34. 2024Modality-Aware Shot Relating and Comparing for Video Scene Detectioncited by 0
  35. 2025First-order State Space Model for Lightweight Image Super-resolutioncited by 0
  36. 2025ReactionMamba: Generating Short & Long Human Reaction Sequencescited by 0
  37. 2026Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QAcited by 0
  38. 2026Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAGcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Selective Structured State-Spaces for Long-Form Video Understanding
  2. 2025Pushing the Boundaries of State Space Models for Image and Video Generation
  3. 2023Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation
  4. 2026Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detection
  5. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
  6. 2024Understanding Video Transformers via Universal Concept Discovery
  7. 2025State Space Model Meets Transformer: A New Paradigm for 3D Object Detection
  8. 2024MAMBA4D: Efficient Long-Sequence Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models