# Spatio-Temporal Dynamic Inference Network for Group Activity Recognition

arXiv: https://arxiv.org/abs/2108.11743 · HTML: https://mlsanity.com/paper/2108.11743/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 54 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs](https://mlsanity.com/paper/1902.10191/) (2019)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [Hierarchical Deep Temporal Models for Group Activity Recognition](https://mlsanity.com/paper/1607.02643/) (2016)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Learning Actor Relation Graphs for Group Activity Recognition](https://mlsanity.com/paper/1904.10117/) (2019)
- [Dynamic Graph Message Passing Networks](https://mlsanity.com/paper/1908.06955/) (2019)
- [VAIN: Attentional Multi-agent Predictive Modeling](https://mlsanity.com/paper/1706.06122/) (2017)
- [Social Scene Understanding: End-to-End Multi-Person Action Localization and Collective Activity Recognition](https://mlsanity.com/paper/1611.09078/) (2016)
- [Social Adaptive Module for Weakly-supervised Group Activity Recognition](https://mlsanity.com/paper/2007.09470/) (2020)

## Built on by (most cited first, 30 of 42)

- [Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation](https://mlsanity.com/paper/2204.02136/) (2022)
- [RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection](https://mlsanity.com/paper/2209.01814/) (2022)
- [RLIPv2: Fast Scaling of Relational Language-Image Pre-training](https://mlsanity.com/paper/2308.09351/) (2023)
- [Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports](https://mlsanity.com/paper/2401.01505/) (2024)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition](https://mlsanity.com/paper/2204.02148/) (2022)
- [SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2410.08474/) (2024)
- [Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics](https://mlsanity.com/paper/2202.00259/) (2022)
- [Hunting Group Clues with Transformers for Social Group Activity Recognition](https://mlsanity.com/paper/2207.05254/) (2022)
- [Detector-Free Weakly Supervised Group Activity Recognition](https://mlsanity.com/paper/2204.02139/) (2022)
- [COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality](https://mlsanity.com/paper/2112.05892/) (2021)
- [SPARTAN: Self-supervised Spatiotemporal Transformers Approach to Group Activity Recognition](https://mlsanity.com/paper/2303.12149/) (2023)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation](https://mlsanity.com/paper/2308.16876/) (2023)
- [Towards More Practical Group Activity Detection: A New Benchmark and Model](https://mlsanity.com/paper/2312.02878/) (2023)
- [EvolveHypergraph: Group-Aware Dynamic Relational Reasoning for Trajectory Prediction](https://mlsanity.com/paper/2208.05470/) (2022)
- [SoGAR: Self-supervised Spatiotemporal Attention-based Social Group Activity Recognition](https://mlsanity.com/paper/2305.06310/) (2023)
- [Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph](https://mlsanity.com/paper/2407.19497/) (2024)
- [FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding](https://mlsanity.com/paper/2504.19514/) (2025)
- [REACT: Recognize Every Action Everywhere All At Once](https://mlsanity.com/paper/2312.00188/) (2023)
- [Flow-Assisted Motion Learning Network for Weakly-Supervised Group Activity Recognition](https://mlsanity.com/paper/2405.18012/) (2024)
- [SGA-INTERACT: A 3D Skeleton-based Benchmark for Group Activity Understanding in Modern Basketball Tactic](https://mlsanity.com/paper/2503.06522/) (2025)
- [CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition](https://mlsanity.com/paper/2410.07153/) (2024)
- [Seeing Through Deepfakes: A Human-Inspired Framework for Multi-Face Detection](https://mlsanity.com/paper/2507.14807/) (2025)
- [Knowledge Augmented Relation Inference for Group Activity Recognition](https://mlsanity.com/paper/2302.14350/) (2023)
- [DECOMPL: Decompositional Learning with Attention Pooling for Group Activity Recognition from a Single Volleyball Image](https://mlsanity.com/paper/2303.06439/) (2023)
- [Learning Group Activity Features Through Person Attribute Prediction](https://mlsanity.com/paper/2403.02753/) (2024)
- [MLP-AIR: An Efficient MLP-Based Method for Actor Interaction Relation Learning in Group Activity Recognition](https://mlsanity.com/paper/2304.08803/) (2023)
- [Spatio-Temporal Proximity-Aware Dual-Path Model for Panoramic Activity Recognition](https://mlsanity.com/paper/2403.14113/) (2024)
- [Design and Analysis of Efficient Attention in Transformers for Social Group Activity Recognition](https://mlsanity.com/paper/2404.09964/) (2024)

## Abstract (arXiv)

Group activity recognition aims to understand the activity performed by a group of people. In order to solve it, modeling complex spatio-temporal interactions is the key. Previous methods are limited in reasoning on a predefined graph, which ignores the inherent person-specific interaction context. Moreover, they adopt inference schemes that are computationally expensive and easily result in the over-smoothing problem. In this paper, we manage to achieve spatio-temporal person-specific inferences by proposing Dynamic Inference Network (DIN), which composes of Dynamic Relation (DR) module and Dynamic Walk (DW) module. We firstly propose to initialize interaction fields on a primary spatio-temporal graph. Within each interaction field, we apply DR to predict the relation matrix and DW to predict the dynamic walk offsets in a joint-processing manner, thus forming a person-specific interaction graph. By updating features on the specific graph, a person can possess a global-level interaction field with a local initialization. Experiments indicate both modules' effectiveness. Moreover, DIN achieves significant improvement compared to previous state-of-the-art methods on two popular datasets under the same setting, while costing much less computation overhead of the reasoning module.