# Pose is all you need: The pose only group activity recognition system (POGARS)

arXiv: https://arxiv.org/abs/2108.04186 · HTML: https://mlsanity.com/paper/2108.04186/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [Hierarchical Deep Temporal Models for Group Activity Recognition](https://mlsanity.com/paper/1607.02643/) (2016)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Learning Actor Relation Graphs for Group Activity Recognition](https://mlsanity.com/paper/1904.10117/) (2019)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks](https://mlsanity.com/paper/1704.02581/) (2017)
- [Social Scene Understanding: End-to-End Multi-Person Action Localization and Collective Activity Recognition](https://mlsanity.com/paper/1611.09078/) (2016)
- [Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos](https://mlsanity.com/paper/2007.02632/) (2020)
- [Convolutional Relational Machine for Group Activity Recognition](https://mlsanity.com/paper/1904.03308/) (2019)
- [Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition](https://mlsanity.com/paper/1511.04196/) (2015)
- [Detecting events and key actors in multi-person videos](https://mlsanity.com/paper/1511.02917/) (2015)
- [CERN: Confidence-Energy Recurrent Network for Group Activity Recognition](https://mlsanity.com/paper/1704.03058/) (2017)
- [Skeleton-based Relational Reasoning for Group Activity Analysis](https://mlsanity.com/paper/2011.05653/) (2020)
- [Action Machine: Rethinking Action Recognition in Trimmed Videos](https://mlsanity.com/paper/1812.05770/) (2018)
- [Deep Keyframe Detection in Human Action Videos](https://mlsanity.com/paper/1804.10021/) (2018)
- [Learning Person Trajectory Representations for Team Activity Analysis](https://mlsanity.com/paper/1706.00893/) (2017)
- [A Multi-Stream Convolutional Neural Network Framework for Group Activity Recognition](https://mlsanity.com/paper/1812.10328/) (2018)
- [Pose-conditioned Spatio-Temporal Attention for Human Action Recognition](https://mlsanity.com/paper/1703.10106/) (2017)

## Built on by (most cited first, 16 of 16)

- [LOGO: A Long-Form Video Dataset for Group Action Quality Assessment](https://mlsanity.com/paper/2404.05029/) (2024)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [A Comprehensive Review of Computer Vision in Sports: Open Issues, Future Trends and Research Directions](https://mlsanity.com/paper/2203.02281/) (2022)
- [COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality](https://mlsanity.com/paper/2112.05892/) (2021)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph](https://mlsanity.com/paper/2407.19497/) (2024)
- [P2ANet: A Dataset and Benchmark for Dense Action Detection from Table Tennis Match Broadcasting Videos](https://mlsanity.com/paper/2207.12730/) (2022)
- [M3Act: Learning from Synthetic Human Group Activities](https://mlsanity.com/paper/2306.16772/) (2023)
- [SGA-INTERACT: A 3D Skeleton-based Benchmark for Group Activity Understanding in Modern Basketball Tactic](https://mlsanity.com/paper/2503.06522/) (2025)
- [CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition](https://mlsanity.com/paper/2410.07153/) (2024)
- [DECOMPL: Decompositional Learning with Attention Pooling for Group Activity Recognition from a Single Volleyball Image](https://mlsanity.com/paper/2303.06439/) (2023)
- [SkeleTR: Towrads Skeleton-based Action Recognition in the Wild](https://mlsanity.com/paper/2309.11445/) (2023)
- [Group Activity Recognition using Unreliable Tracked Pose](https://mlsanity.com/paper/2401.03262/) (2024)
- [PromptGAR: Flexible Promptive Group Activity Recognition](https://mlsanity.com/paper/2503.08933/) (2025)
- [Group-DINOmics: Incorporating People Dynamics into DINO for Self-supervised Group Activity Feature Learning](https://mlsanity.com/paper/2604.04467/) (2026)
- [De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift](https://mlsanity.com/paper/2609.32454/) (2026)

## Abstract (arXiv)

We introduce a novel deep learning based group activity recognition approach called the Pose Only Group Activity Recognition System (POGARS), designed to use only tracked poses of people to predict the performed group activity. In contrast to existing approaches for group activity recognition, POGARS uses 1D CNNs to learn spatiotemporal dynamics of individuals involved in a group activity and forgo learning features from pixel data. The proposed model uses a spatial and temporal attention mechanism to infer person-wise importance and multi-task learning for simultaneously performing group and individual action classification. Experimental results confirm that POGARS achieves highly competitive results compared to state-of-the-art methods on a widely used public volleyball dataset despite only using tracked pose as input. Further our experiments show by using pose only as input, POGARS has better generalization capabilities compared to methods that use RGB as input.