# Learning Group Activities from Skeletons without Individual Action Labels

arXiv: https://arxiv.org/abs/2105.06754 · HTML: https://mlsanity.com/paper/2105.06754/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 14 May 2021.

## Builds on (most cited first, 16 of 16)

- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [Hierarchical Deep Temporal Models for Group Activity Recognition](https://mlsanity.com/paper/1607.02643/) (2016)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Learning Actor Relation Graphs for Group Activity Recognition](https://mlsanity.com/paper/1904.10117/) (2019)
- [P-CNN: Pose-based CNN Features for Action Recognition](https://mlsanity.com/paper/1506.03607/) (2015)
- [Social Scene Understanding: End-to-End Multi-Person Action Localization and Collective Activity Recognition](https://mlsanity.com/paper/1611.09078/) (2016)
- [Convolutional Relational Machine for Group Activity Recognition](https://mlsanity.com/paper/1904.03308/) (2019)
- [Progressive Relation Learning for Group Activity Recognition](https://mlsanity.com/paper/1908.02948/) (2019)
- [Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition](https://mlsanity.com/paper/1511.04196/) (2015)
- [CERN: Confidence-Energy Recurrent Network for Group Activity Recognition](https://mlsanity.com/paper/1704.03058/) (2017)
- [Visual Recognition by Counting Instances: A Multi-Instance Cardinality Potential Kernel](https://mlsanity.com/paper/1502.02063/) (2015)
- [A Multi-Stream Convolutional Neural Network Framework for Group Activity Recognition](https://mlsanity.com/paper/1812.10328/) (2018)

## Built on by (most cited first, 4 of 4)

- [COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality](https://mlsanity.com/paper/2112.05892/) (2021)
- [Group Activity Recognition in Basketball Tracking Data -- Neural Embeddings in Team Sports (NETS)](https://mlsanity.com/paper/2209.00451/) (2022)
- [M3Act: Learning from Synthetic Human Group Activities](https://mlsanity.com/paper/2306.16772/) (2023)
- [DECOMPL: Decompositional Learning with Attention Pooling for Group Activity Recognition from a Single Volleyball Image](https://mlsanity.com/paper/2303.06439/) (2023)

## Abstract (arXiv)

To understand human behavior we must not just recognize individual actions but model possibly complex group activity and interactions. Hierarchical models obtain the best results in group activity recognition but require fine grained individual action annotations at the actor level. In this paper we show that using only skeletal data we can train a state-of-the art end-to-end system using only group activity labels at the sequence level. Our experiments show that models trained without individual action supervision perform poorly. On the other hand we show that pseudo-labels can be computed from any pre-trained feature extractor with comparable final performance. Finally our carefully designed lean pose only architecture shows highly competitive results versus more complex multimodal approaches even in the self-supervised variant.