# Detecting events and key actors in multi-person videos

arXiv: https://arxiv.org/abs/1511.02917 · HTML: https://mlsanity.com/paper/1511.02917/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2015.

## Built on by (most cited first, 26 of 26)

- [Faster VoxelPose: Real-time 3D Human Pose Estimation by Orthographic Projection](https://mlsanity.com/paper/2207.10955/) (2022)
- [LOGO: A Long-Form Video Dataset for Group Action Quality Assessment](https://mlsanity.com/paper/2404.05029/) (2024)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [A Comprehensive Review of Computer Vision in Sports: Open Issues, Future Trends and Research Directions](https://mlsanity.com/paper/2203.02281/) (2022)
- [COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality](https://mlsanity.com/paper/2112.05892/) (2021)
- [Pose is all you need: The pose only group activity recognition system (POGARS)](https://mlsanity.com/paper/2108.04186/) (2021)
- [Hang-Time HAR: A Benchmark Dataset for Basketball Activity Recognition using Wrist-Worn Inertial Sensors](https://mlsanity.com/paper/2305.13124/) (2023)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [Most Important Person-guided Dual-branch Cross-Patch Attention for Group Affect Recognition](https://mlsanity.com/paper/2212.07055/) (2022)
- [Evaluating Soccer Player: from Live Camera to Deep Reinforcement Learning](https://mlsanity.com/paper/2101.05388/) (2021)
- [A Unified Taxonomy and Multimodal Dataset for Events in Invasion Games](https://mlsanity.com/paper/2108.11149/) (2021)
- [SGA-INTERACT: A 3D Skeleton-based Benchmark for Group Activity Understanding in Modern Basketball Tactic](https://mlsanity.com/paper/2503.06522/) (2025)
- [EventAnchor: Reducing Human Interactions in Event Annotation of Racket Sports Videos](https://mlsanity.com/paper/2101.04954/) (2021)
- [Real-time Aerial Detection and Reasoning on Embedded-UAVs](https://mlsanity.com/paper/2305.12414/) (2023)
- [MIP-GAF: A MLLM-annotated Benchmark for Most Important Person Localization and Group Context Understanding](https://mlsanity.com/paper/2409.06224/) (2024)
- [Deep Learning for Sports Video Event Detection: Tasks, Datasets, Methods, and Challenges](https://mlsanity.com/paper/2505.03991/) (2025)
- [Distantly Supervised Semantic Text Detection and Recognition for Broadcast Sports Videos Understanding](https://mlsanity.com/paper/2111.00629/) (2021)
- [pix2pockets: Shot Suggestions in 8-Ball Pool from a Single Image in the Wild](https://mlsanity.com/paper/2504.12045/) (2025)
- [Hands Off: A Handshake Interaction Detection and Localization Model for COVID-19 Threat Control](https://mlsanity.com/paper/2110.09571/) (2021)
- [MLP-AIR: An Efficient MLP-Based Method for Actor Interaction Relation Learning in Group Activity Recognition](https://mlsanity.com/paper/2304.08803/) (2023)
- [Group Activity Recognition using Unreliable Tracked Pose](https://mlsanity.com/paper/2401.03262/) (2024)
- [Towards Comprehensive Basketball Understanding](https://mlsanity.com/paper/2608.23435/) (2026)
- [Discovering Multi-Label Actor-Action Association in a Weakly Supervised Setting](https://mlsanity.com/paper/2101.08567/) (2021)
- [Group Activity Recognition in Computer Vision: A Comprehensive Review, Challenges, and Future Perspectives](https://mlsanity.com/paper/2307.13541/) (2023)
- [Calisthenics Skills Temporal Video Segmentation](https://mlsanity.com/paper/2507.12245/) (2025)

## Abstract (arXiv)

Multi-person event recognition is a challenging task, often with many people active in the scene but only a small subset contributing to an actual event. In this paper, we propose a model which learns to detect events in such videos while automatically "attending" to the people responsible for the event. Our model does not use explicit annotations regarding who or where those people are during training and testing. In particular, we track people in videos and use a recurrent neural network (RNN) to represent the track features. We learn time-varying attention weights to combine these features at each time-instant. The attended features are then processed using another RNN for event detection/classification. Since most video datasets with multiple people are restricted to a small number of videos, we also collected a new basketball dataset comprising 257 basketball games with 14K event annotations corresponding to 11 event classes. Our model outperforms state-of-the-art methods for both event classification and detection on this new dataset. Additionally, we show that the attention mechanism is able to consistently localize the relevant players.