# Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detection

arXiv: https://arxiv.org/abs/2604.09164 · HTML: https://mlsanity.com/paper/2604.09164/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [Graph Mamba: Towards Learning on Graphs with State Space Models](https://mlsanity.com/paper/2402.08678/) (2024)
- [End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames](https://mlsanity.com/paper/2311.17241/) (2023)
- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces](https://mlsanity.com/paper/2503.06157/) (2025)
- [VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI](https://mlsanity.com/paper/2410.11623/) (2024)
- [TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localization](https://mlsanity.com/paper/2303.09055/) (2023)
- [AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding](https://mlsanity.com/paper/2406.13807/) (2024)
- [ASTRA: An Action Spotting TRAnsformer for Soccer Videos](https://mlsanity.com/paper/2404.01891/) (2024)
- [ETAD: Training Action Detection End to End on a Laptop](https://mlsanity.com/paper/2205.07134/) (2022)
- [Low-power, Continuous Remote Behavioral Localization with Event Cameras](https://mlsanity.com/paper/2312.03799/) (2023)
- [MambaTAD: When State-Space Models Meet Long-Range Temporal Action Detection](https://mlsanity.com/paper/2511.17929/) (2025)
- [Temporal Action Localization with Cross Layer Task Decoupling and Refinement](https://mlsanity.com/paper/2412.09202/) (2024)

## Abstract (arXiv)

Temporal human action detection aims to identify and localize action segments within untrimmed videos, serving as a pivotal task in video understanding. Despite the progress achieved by prior architectures like CNN and Transformer models, these continue to struggle with feature redundancy and degraded global dependency modeling capabilities when applied to long video sequences. These limitations severely constrain their scalability in real-world video analysis. State Space Models (SSMs) offer a promising alternative with linear long-term modeling and robust global temporal reasoning capabilities. Rethinking the application of SSMs in temporal modeling, this research constructs a novel framework for video human action detection. Specifically, we introduce the Efficient Spatial-Temporal Focal (ESTF) Adapter into the pre-trained layers. This module integrates the advantages of our proposed Temporal Boundary-aware SSM(TB-SSM) for temporal feature modeling with efficient processing of spatial features. We perform comprehensive and quantitative analyses across multiple benchmarks, comparing our proposed method against previous SSM-based and other structural methods. Extensive experiments demonstrate that our improved strategy significantly enhances both localization performance and robustness, validating the effectiveness of our proposed method.