# PKS4:Parallel Kinematic Selective State Space Scanners for Efficient Video Understanding

arXiv: https://arxiv.org/abs/2604.26461 · HTML: https://mlsanity.com/paper/2604.26461/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 29 Apr 2026.

## Placed in

- State-space sequence models (S4, Mamba) / [Video, motion and generative Mamba models](https://mlsanity.com/ssm/video-motion-and-generative-mamba-models/)

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning](https://mlsanity.com/paper/2201.04676/) (2022)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [AIM: Adapting Image Models for Efficient Video Action Recognition](https://mlsanity.com/paper/2302.03024/) (2023)
- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [Revisiting Classifier: Transferring Vision-Language Models for Video Recognition](https://mlsanity.com/paper/2207.01297/) (2022)
- [Token Shift Transformer for Video Classification](https://mlsanity.com/paper/2108.02432/) (2021)
- [Vivim: a Video Vision Mamba for Medical Video Segmentation](https://mlsanity.com/paper/2401.14168/) (2024)
- [Dual-path Adaptation from Image to Video Transformers](https://mlsanity.com/paper/2303.09857/) (2023)
- [Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2309.07911/) (2023)

## Abstract (arXiv)

Temporal modeling remains a fundamental challenge in video understanding, particularly as sequence lengths scale. Traditional video models relying on dense spatiotemporal attention suffer from quadratic computational costs for long videos. To circumvent these costs, recent approaches adapt image models for videos via Parameter-Efficient Fine-Tuning (PEFT) methods such as adapters. However, deeply inserting these modules incurs prohibitive activation memory overhead during back-propagation. While recent efficient State Space Models (SSMs) introduce linear complexity, they disrupt 2D spatial relationships and rely on extensive masked pre-training to recover spatial awareness. To overcome these limitations, we propose Parallel Kinematic Selective State Space Scanners (PKS4). We retain a standard 2D vision backbone for spatial semantics and insert a single plug-and-play PKS4 module with linear-complexity temporal scanning, avoiding temporal attention and multi-layer adapters. We first extract kinematic priors via a Kinematic Prior Encoder, which captures local displacements and motion boundaries through inter-frame correlations and differences. These priors drive linear-complexity SSMs to track underlying kinematic states, adaptively modulating update speeds and read-write strategies at each time step. Instead of global scanning, we deploy parallel scanners along the temporal dimension for each spatial location, preserving spatial structures while reducing overhead. Experiments on spatial-heavy and temporal-heavy action recognition benchmarks show that PKS4 achieves state-of-the-art performance. Remarkably, our method converges in merely 20 epochs, achieving approximately 10× lower training compute than pure video SSMs, establishing a new paradigm for efficient video understanding.