# SPARTAN: Self-supervised Spatiotemporal Transformers Approach to Group Activity Recognition

arXiv: https://arxiv.org/abs/2303.12149 · HTML: https://mlsanity.com/paper/2303.12149/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 72 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [Hierarchical Deep Temporal Models for Group Activity Recognition](https://mlsanity.com/paper/1607.02643/) (2016)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Learning Actor Relation Graphs for Group Activity Recognition](https://mlsanity.com/paper/1904.10117/) (2019)

## Built on by (most cited first, 14 of 14)

- [Tracking Skiers from the Top to the Bottom](https://mlsanity.com/paper/2312.09723/) (2023)
- [GSON: A Group-based Social Navigation Framework with Large Multimodal Model](https://mlsanity.com/paper/2409.18084/) (2024)
- [REACT: Recognize Every Action Everywhere All At Once](https://mlsanity.com/paper/2312.00188/) (2023)
- [HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos](https://mlsanity.com/paper/2312.07740/) (2023)
- [Flow-Assisted Motion Learning Network for Weakly-Supervised Group Activity Recognition](https://mlsanity.com/paper/2405.18012/) (2024)
- [FLAASH: Flow-Attention Adaptive Semantic Hierarchical Fusion for Multi-Modal Tobacco Content Analysis](https://mlsanity.com/paper/2410.19896/) (2024)
- [Public Health Advocacy Dataset: A Dataset of Tobacco Usage Videos from Social Media](https://mlsanity.com/paper/2411.13572/) (2024)
- [Multi-view Action Recognition via Directed Gromov-Wasserstein Discrepancy](https://mlsanity.com/paper/2405.01337/) (2024)
- [MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition](https://mlsanity.com/paper/2408.00420/) (2024)
- [High-Order Evolving Graphs for Enhanced Representation of Traffic Dynamics](https://mlsanity.com/paper/2409.11206/) (2024)
- [PromptGAR: Flexible Promptive Group Activity Recognition](https://mlsanity.com/paper/2503.08933/) (2025)
- [VIFSS: View-Invariant and Figure Skating-Specific Pose Representation Learning for Temporal Action Segmentation](https://mlsanity.com/paper/2508.10281/) (2025)
- [Group-DINOmics: Incorporating People Dynamics into DINO for Self-supervised Group Activity Feature Learning](https://mlsanity.com/paper/2604.04467/) (2026)
- [Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning](https://mlsanity.com/paper/2607.23077/) (2026)

## Abstract (arXiv)

In this paper, we propose a new, simple, and effective Self-supervised Spatio-temporal Transformers (SPARTAN) approach to Group Activity Recognition (GAR) using unlabeled video data. Given a video, we create local and global Spatio-temporal views with varying spatial patch sizes and frame rates. The proposed self-supervised objective aims to match the features of these contrasting views representing the same video to be consistent with the variations in spatiotemporal domains. To the best of our knowledge, the proposed mechanism is one of the first works to alleviate the weakly supervised setting of GAR using the encoders in video transformers. Furthermore, using the advantage of transformer models, our proposed approach supports long-term relationship modeling along spatio-temporal dimensions. The proposed SPARTAN approach performs well on two group activity recognition benchmarks, including NBA and Volleyball datasets, by surpassing the state-of-the-art results by a significant margin in terms of MCA and MPCA metrics.