# Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition

arXiv: https://arxiv.org/abs/1511.04196 · HTML: https://mlsanity.com/paper/1511.04196/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2015.

## Built on by (most cited first, 27 of 27)

- [GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer](https://mlsanity.com/paper/2108.12630/) (2021)
- [Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition](https://mlsanity.com/paper/2204.02148/) (2022)
- [When Pigs Fly: Contextual Reasoning in Synthetic and Natural Scenes](https://mlsanity.com/paper/2104.02215/) (2021)
- [Hunting Group Clues with Transformers for Social Group Activity Recognition](https://mlsanity.com/paper/2207.05254/) (2022)
- [Detector-Free Weakly Supervised Group Activity Recognition](https://mlsanity.com/paper/2204.02139/) (2022)
- [COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality](https://mlsanity.com/paper/2112.05892/) (2021)
- [Pose is all you need: The pose only group activity recognition system (POGARS)](https://mlsanity.com/paper/2108.04186/) (2021)
- [SPARTAN: Self-supervised Spatiotemporal Transformers Approach to Group Activity Recognition](https://mlsanity.com/paper/2303.12149/) (2023)
- [Human Interaction Recognition Framework based on Interacting Body Part Attention](https://mlsanity.com/paper/2101.08967/) (2021)
- [Towards More Practical Group Activity Detection: A New Benchmark and Model](https://mlsanity.com/paper/2312.02878/) (2023)
- [Representing Videos as Discriminative Sub-graphs for Action Recognition](https://mlsanity.com/paper/2201.04027/) (2022)
- [SoGAR: Self-supervised Spatiotemporal Attention-based Social Group Activity Recognition](https://mlsanity.com/paper/2305.06310/) (2023)
- [Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph](https://mlsanity.com/paper/2407.19497/) (2024)
- [REACT: Recognize Every Action Everywhere All At Once](https://mlsanity.com/paper/2312.00188/) (2023)
- [Learning Group Activities from Skeletons without Individual Action Labels](https://mlsanity.com/paper/2105.06754/) (2021)
- [HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos](https://mlsanity.com/paper/2312.07740/) (2023)
- [Flow-Assisted Motion Learning Network for Weakly-Supervised Group Activity Recognition](https://mlsanity.com/paper/2405.18012/) (2024)
- [Temporal and Object Quantification Networks](https://mlsanity.com/paper/2106.05891/) (2021)
- [Learning to See the Elephant in the Room: Self-Supervised Context Reasoning in Humans and AI](https://mlsanity.com/paper/2211.12817/) (2022)
- [Free-Form Composition Networks for Egocentric Action Recognition](https://mlsanity.com/paper/2307.06527/) (2023)
- [Design and Analysis of Efficient Attention in Transformers for Social Group Activity Recognition](https://mlsanity.com/paper/2404.09964/) (2024)
- [LiGAR: LiDAR-Guided Hierarchical Transformer for Multi-Modal Group Activity Recognition](https://mlsanity.com/paper/2410.21108/) (2024)
- [Context-driven Visual Object Recognition based on Knowledge Graphs](https://mlsanity.com/paper/2210.11233/) (2022)
- [Group Activity Recognition via Dynamic Composition and Interaction](https://mlsanity.com/paper/2305.05583/) (2023)
- [Group Activity Recognition in Computer Vision: A Comprehensive Review, Challenges, and Future Perspectives](https://mlsanity.com/paper/2307.13541/) (2023)
- [VicKAM: Visual Conceptual Knowledge Guided Action Map for Weakly Supervised Group Activity Recognition](https://mlsanity.com/paper/2502.09967/) (2025)
- [Multi-Focused Video Group Activities Hashing](https://mlsanity.com/paper/2509.00490/) (2025)

## Abstract (arXiv)

Rich semantic relations are important in a variety of visual recognition problems. As a concrete example, group activity recognition involves the interactions and relative spatial relations of a set of people in a scene. State of the art recognition methods center on deep learning approaches for training highly effective, complex classifiers for interpreting images. However, bridging the relatively low-level concepts output by these methods to interpret higher-level compositional scenes remains a challenge. Graphical models are a standard tool for this task. In this paper, we propose a method to integrate graphical models and deep neural networks into a joint framework. Instead of using a traditional inference method, we use a sequential inference modeled by a recurrent neural network. Beyond this, the appropriate structure for inference can be learned by imposing gates on edges between nodes. Empirical results on group activity recognition demonstrate the potential of this model to handle highly structured learning tasks.