# Hierarchical Deep Temporal Models for Group Activity Recognition

arXiv: https://arxiv.org/abs/1607.02643 · HTML: https://mlsanity.com/paper/1607.02643/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2016.

## Built on by (most cited first, 30 of 89)

- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer](https://mlsanity.com/paper/2108.12630/) (2021)
- [AGENT: A Benchmark for Core Psychological Reasoning](https://mlsanity.com/paper/2102.12321/) (2021)
- [Spatio-Temporal Dynamic Inference Network for Group Activity Recognition](https://mlsanity.com/paper/2108.11743/) (2021)
- [Spotting Temporally Precise, Fine-Grained Events in Video](https://mlsanity.com/paper/2207.10213/) (2022)
- [JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection](https://mlsanity.com/paper/2106.08827/) (2021)
- [LOGO: A Long-Form Video Dataset for Group Action Quality Assessment](https://mlsanity.com/paper/2404.05029/) (2024)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition](https://mlsanity.com/paper/2204.02148/) (2022)
- [CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process](https://mlsanity.com/paper/2401.14535/) (2024)
- [PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception](https://mlsanity.com/paper/2103.01933/) (2021)
- [Hunting Group Clues with Transformers for Social Group Activity Recognition](https://mlsanity.com/paper/2207.05254/) (2022)
- [A Comprehensive Review of Computer Vision in Sports: Open Issues, Future Trends and Research Directions](https://mlsanity.com/paper/2203.02281/) (2022)
- [Detector-Free Weakly Supervised Group Activity Recognition](https://mlsanity.com/paper/2204.02139/) (2022)
- [COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality](https://mlsanity.com/paper/2112.05892/) (2021)
- [Pose is all you need: The pose only group activity recognition system (POGARS)](https://mlsanity.com/paper/2108.04186/) (2021)
- [SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos](https://mlsanity.com/paper/2404.04565/) (2024)
- [SPARTAN: Self-supervised Spatiotemporal Transformers Approach to Group Activity Recognition](https://mlsanity.com/paper/2303.12149/) (2023)
- [MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations](https://mlsanity.com/paper/2410.13790/) (2024)
- [Video Action Differencing](https://mlsanity.com/paper/2503.07860/) (2025)
- [Panoramic Human Activity Recognition](https://mlsanity.com/paper/2203.03806/) (2022)
- [Causal Temporal Representation Learning with Nonstationary Sparse Transition](https://mlsanity.com/paper/2409.03142/) (2024)
- [Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes](https://mlsanity.com/paper/2311.17948/) (2023)
- [Human Interaction Recognition Framework based on Interacting Body Part Attention](https://mlsanity.com/paper/2101.08967/) (2021)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Towards More Practical Group Activity Detection: A New Benchmark and Model](https://mlsanity.com/paper/2312.02878/) (2023)
- [SoccerMaster: A Vision Foundation Model for Soccer Understanding](https://mlsanity.com/paper/2512.11016/) (2025)
- [Widely Applicable Strong Baseline for Sports Ball Detection and Tracking](https://mlsanity.com/paper/2311.05237/) (2023)
- [A Survey of Video Datasets for Grounded Event Understanding](https://mlsanity.com/paper/2406.09646/) (2024)
- [SoGAR: Self-supervised Spatiotemporal Attention-based Social Group Activity Recognition](https://mlsanity.com/paper/2305.06310/) (2023)

## Abstract (arXiv)

In this paper we present an approach for classifying the activity performed by a group of people in a video sequence. This problem of group activity recognition can be addressed by examining individual person actions and their relations. Temporal dynamics exist both at the level of individual person actions as well as at the level of group activity. Given a video sequence as input, methods can be developed to capture these dynamics at both person-level and group-level detail. We build a deep model to capture these dynamics based on LSTM (long short-term memory) models. In order to model both person-level and group-level dynamics, we present a 2-stage deep temporal model for the group activity recognition problem. In our approach, one LSTM model is designed to represent action dynamics of individual people in a video sequence and another LSTM model is designed to aggregate person-level information for group activity recognition. We collected a new dataset consisting of volleyball videos labeled with individual and group activities in order to evaluate our method. Experimental results on this new Volleyball Dataset and the standard benchmark Collective Activity Dataset demonstrate the efficacy of the proposed models.