# Game State and Spatio-temporal Action Detection in Soccer using Graph Neural Networks and 3D Convolutional Networks

arXiv: https://arxiv.org/abs/2502.15462 · HTML: https://mlsanity.com/paper/2502.15462/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [Action Tubelet Detector for Spatio-Temporal Action Localization](https://mlsanity.com/paper/1705.01861/) (2017)
- [You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization](https://mlsanity.com/paper/1911.06644/) (2019)
- [Online Real-time Multiple Spatiotemporal Action Localisation and Prediction](https://mlsanity.com/paper/1611.08563/) (2016)
- [Actions as Moving Points](https://mlsanity.com/paper/2001.04608/) (2020)
- [TubeR: Tubelet Transformer for Video Action Detection](https://mlsanity.com/paper/2104.00969/) (2021)
- [Learning to track for spatio-temporal action localization](https://mlsanity.com/paper/1506.01929/) (2015)
- [Social Scene Understanding: End-to-End Multi-Person Action Localization and Collective Activity Recognition](https://mlsanity.com/paper/1611.09078/) (2016)
- [Artificial Intelligence and Statistics](https://mlsanity.com/paper/1712.03779/) (2017)
- [A Structured Model For Action Detection](https://mlsanity.com/paper/1812.03544/) (2018)
- [Spotting Temporally Precise, Fine-Grained Events in Video](https://mlsanity.com/paper/2207.10213/) (2022)
- [Deep Learning for Detecting Multiple Space-Time Action Tubes in Videos](https://mlsanity.com/paper/1608.01529/) (2016)
- [Convolutional Relational Machine for Group Activity Recognition](https://mlsanity.com/paper/1904.03308/) (2019)
- [SoccerNet 2023 Challenges Results](https://mlsanity.com/paper/2309.06006/) (2023)
- [A Better Baseline for AVA](https://mlsanity.com/paper/1807.10066/) (2018)
- [Spatio-Temporal Action Detection Under Large Motion](https://mlsanity.com/paper/2209.02250/) (2022)
- [A Graph-Based Method for Soccer Action Spotting Using Unsupervised Player Classification](https://mlsanity.com/paper/2211.12334/) (2022)

## Built on by (most cited first, 6 of 6)

- [Beyond Pixels: Leveraging the Language of Soccer to Improve Spatio-Temporal Action Detection in Broadcast Videos](https://mlsanity.com/paper/2505.09455/) (2025)
- [FOOTPASS: A Multi-Modal Multi-Agent Tactical Context Dataset for Play-by-Play Action Spotting in Soccer Broadcast Videos](https://mlsanity.com/paper/2511.16183/) (2025)
- [Multi-Modal Soccer Scene Analysis with Masked Pre-Training](https://mlsanity.com/paper/2512.19528/) (2025)
- [SoccerNet 2026 Challenges Results](https://mlsanity.com/paper/2607.07320/) (2026)
- [SoccerNet 2026 Player-Centric Ball-Action Spotting:Retraining and Post-Processing Extensions to the FOOTPASS Baselines](https://mlsanity.com/paper/2606.09679/) (2026)
- [Entity-Aware Sequence Transduction for Player-Centric Ball Action Spotting](https://mlsanity.com/paper/2608.01696/) (2026)

## Abstract (arXiv)

Soccer analytics rely on two data sources: the player positions on the pitch and the sequences of events they perform. With around 2000 ball events per game, their precise and exhaustive annotation based on a monocular video stream remains a tedious and costly manual task. While state-of-the-art spatio-temporal action detection methods show promise for automating this task, they lack contextual understanding of the game. Assuming professional players' behaviors are interdependent, we hypothesize that incorporating surrounding players' information such as positions, velocity and team membership can enhance purely visual predictions. We propose a spatio-temporal action detection approach that combines visual and game state information via Graph Neural Networks trained end-to-end with state-of-the-art 3D CNNs, demonstrating improved metrics through game state integration.