# STMixer: A One-Stage Sparse Action Detector

arXiv: https://arxiv.org/abs/2303.15879 · HTML: https://mlsanity.com/paper/2303.15879/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 55 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021)
- [Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization](https://mlsanity.com/paper/2006.07976/) (2020)

## Built on by (most cited first, 22 of 22)

- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)
- [SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos](https://mlsanity.com/paper/2404.04565/) (2024)
- [Semi-supervised Active Learning for Video Action Detection](https://mlsanity.com/paper/2312.07169/) (2023)
- [Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection](https://mlsanity.com/paper/2411.10922/) (2024)
- [On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes](https://mlsanity.com/paper/2410.19553/) (2024)
- [Stable Mean Teacher for Semi-supervised Video Action Detection](https://mlsanity.com/paper/2412.07072/) (2024)
- [Open-Vocabulary Spatio-Temporal Action Detection](https://mlsanity.com/paper/2405.10832/) (2024)
- [Multiscale Vision Transformers meet Bipartite Matching for efficient single-stage Action Localization](https://mlsanity.com/paper/2312.17686/) (2023)
- [Spatio-Temporal Context Prompting for Zero-Shot Action Detection](https://mlsanity.com/paper/2408.15996/) (2024)
- [SFMViT: SlowFast Meet ViT in Chaotic World](https://mlsanity.com/paper/2404.16609/) (2024)
- [AdaFPP: Adapt-Focused Bi-Propagating Prototype Learning for Panoramic Activity Recognition](https://mlsanity.com/paper/2405.02538/) (2024)
- [Gaseous Object Detection](https://mlsanity.com/paper/2502.12415/) (2025)
- [Multi-Granularity Hand Action Detection](https://mlsanity.com/paper/2306.10858/) (2023)
- [Action Recognition Utilizing YGAR Dataset](https://mlsanity.com/paper/2310.00831/) (2023)
- [STMixer: A One-Stage Sparse Action Detector](https://mlsanity.com/paper/2404.09842/) (2024)
- [Classification Matters: Improving Video Action Detection with Class-Specific Attention](https://mlsanity.com/paper/2407.19698/) (2024)
- [JARViS: Detecting Actions in Video Using Unified Actor-Scene Context Relation Modeling](https://mlsanity.com/paper/2408.03612/) (2024)
- [Query matching for spatio-temporal action detection with query-based object detector](https://mlsanity.com/paper/2409.18408/) (2024)
- [JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts](https://mlsanity.com/paper/2412.13708/) (2024)
- [Scaling Open-Vocabulary Action Detection](https://mlsanity.com/paper/2504.03096/) (2025)
- [Towards an Effective Action-Region Tracking Framework for Fine-grained Video Action Recognition](https://mlsanity.com/paper/2511.21202/) (2025)
- [VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding](https://mlsanity.com/paper/2609.15606/) (2026)

## Abstract (arXiv)

Traditional video action detectors typically adopt the two-stage pipeline, where a person detector is first employed to generate actor boxes and then 3D RoIAlign is used to extract actor-specific features for classification. This detection paradigm requires multi-stage training and inference, and cannot capture context information outside the bounding box. Recently, a few query-based action detectors are proposed to predict action instances in an end-to-end manner. However, they still lack adaptability in feature sampling and decoding, thus suffering from the issues of inferior performance or slower convergence. In this paper, we propose a new one-stage sparse action detector, termed STMixer. STMixer is based on two core designs. First, we present a query-based adaptive feature sampling module, which endows our STMixer with the flexibility of mining a set of discriminative features from the entire spatiotemporal domain. Second, we devise a dual-branch feature mixing module, which allows our STMixer to dynamically attend to and mix video features along the spatial and the temporal dimension respectively for better feature decoding. Coupling these two designs with a video backbone yields an efficient end-to-end action detector. Without bells and whistles, our STMixer obtains the state-of-the-art results on the datasets of AVA, UCF101-24, and JHMDB.