# SFMViT: SlowFast Meet ViT in Chaotic World

arXiv: https://arxiv.org/abs/2404.16609 · HTML: https://mlsanity.com/paper/2404.16609/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization](https://mlsanity.com/paper/2006.07976/) (2020)
- [Actor-Centric Relation Network](https://mlsanity.com/paper/1807.10982/) (2018)
- [Asynchronous Interaction Aggregation for Action Detection](https://mlsanity.com/paper/2004.07485/) (2020)
- [TubeR: Tubelet Transformer for Video Action Detection](https://mlsanity.com/paper/2104.00969/) (2021)
- [WiderPerson: A Diverse Dataset for Dense Pedestrian Detection in the Wild](https://mlsanity.com/paper/1909.12118/) (2019)
- [STMixer: A One-Stage Sparse Action Detector](https://mlsanity.com/paper/2303.15879/) (2023)
- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)

## Built on by (most cited first, 2 of 2)

- [Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2407.15706/) (2024)
- [DAP: Doppler-aware Point Network for Heterogeneous mmWave Action Recognition](https://mlsanity.com/paper/2605.09604/) (2026)

## Abstract (arXiv)

The task of spatiotemporal action localization in chaotic scenes is a challenging task toward advanced video understanding. Paving the way with high-quality video feature extraction and enhancing the precision of detector-predicted anchors can effectively improve model performance. To this end, we propose a high-performance dual-stream spatiotemporal feature extraction network SFMViT with an anchor pruning strategy. The backbone of our SFMViT is composed of ViT and SlowFast with prior knowledge of spatiotemporal action localization, which fully utilizes ViT's excellent global feature extraction capabilities and SlowFast's spatiotemporal sequence modeling capabilities. Secondly, we introduce the confidence maximum heap to prune the anchors detected in each frame of the picture to filter out the effective anchors. These designs enable our SFMViT to achieve a mAP of 26.62% in the Chaotic World dataset, far exceeding existing models. Code is available at https://github.com/jfightyr/SlowFast-Meet-ViT.