# FOOTPASS: A Multi-Modal Multi-Agent Tactical Context Dataset for Play-by-Play Action Spotting in Soccer Broadcast Videos

arXiv: https://arxiv.org/abs/2511.16183 · HTML: https://mlsanity.com/paper/2511.16183/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 24 of 24)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos](https://mlsanity.com/paper/1804.04527/) (2018)
- [SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos](https://mlsanity.com/paper/2011.13367/) (2020)
- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark in Soccer Videos](https://mlsanity.com/paper/2204.06918/) (2022)
- [TacticAI: an AI assistant for football tactics](https://mlsanity.com/paper/2310.10553/) (2023)
- [Spotting Temporally Precise, Fine-Grained Events in Video](https://mlsanity.com/paper/2207.10213/) (2022)
- [Spatio-Temporal Action Detection Under Large Motion](https://mlsanity.com/paper/2209.02250/) (2022)
- [A Graph-Based Method for Soccer Action Spotting Using Unsupervised Player Classification](https://mlsanity.com/paper/2211.12334/) (2022)
- [FootBots: A Transformer-based Architecture for Motion Prediction in Soccer](https://mlsanity.com/paper/2406.19852/) (2024)
- [Fine-Grained Retrieval of Sports Plays using Tree-Based Alignment of Trajectories](https://mlsanity.com/paper/1710.02255/) (2017)
- [Game State and Spatio-temporal Action Detection in Soccer using Graph Neural Networks and 3D Convolutional Networks](https://mlsanity.com/paper/2502.15462/) (2025)
- [A Survey on Deep Learning-based Spatio-temporal Action Detection](https://mlsanity.com/paper/2308.01618/) (2023)
- [Beyond Pixels: Leveraging the Language of Soccer to Improve Spatio-Temporal Action Detection in Broadcast Videos](https://mlsanity.com/paper/2505.09455/) (2025)
- [Common Data Format (CDF): A Standardized Format for Match-Data in Football (Soccer)](https://mlsanity.com/paper/2505.15820/) (2025)
- [Distinguishing Between Roles of Football Players in Play-by-play Match Event Data](https://mlsanity.com/paper/1809.05173/) (2018)
- [SoccerNet-v3D: Leveraging Sports Broadcast Replays for 3D Scene Understanding](https://mlsanity.com/paper/2504.10106/) (2025)
- [Space evaluation at the starting point of soccer transitions](https://mlsanity.com/paper/2505.14711/) (2025)
- [MVP-Shapley: Feature-based Modeling for Evaluating the Most Valuable Player in Basketball](https://mlsanity.com/paper/2506.04602/) (2025)

## Built on by (most cited first, 5 of 5)

- [SoccerNet 2026 Challenges Results](https://mlsanity.com/paper/2607.07320/) (2026)
- [Training-Free Off-Screen Player Imputation for Broadcast-Based Spatial Football Analytics](https://mlsanity.com/paper/2607.11548/) (2026)
- [SoccerNet 2026 Player-Centric Ball-Action Spotting:Retraining and Post-Processing Extensions to the FOOTPASS Baselines](https://mlsanity.com/paper/2606.09679/) (2026)
- [SoccerNet 2026 Player-Centric Ball Action Spotting: Per-Player Attention with Agreement-Based Ensembling](https://mlsanity.com/paper/2606.28389/) (2026)
- [Entity-Aware Sequence Transduction for Player-Centric Ball Action Spotting](https://mlsanity.com/paper/2608.01696/) (2026)

## Abstract (arXiv)

Soccer video understanding has motivated the creation of datasets for tasks such as temporal action localization, spatiotemporal action detection (STAD), or multiobject tracking (MOT). The annotation of structured sequences of events (who does what, when, and where) used for soccer analytics requires a holistic approach that integrates both STAD and MOT. However, current action recognition methods remain insufficient for constructing reliable play-by-play data and are typically used to assist rather than fully automate annotation. Parallel research has advanced tactical modeling, trajectory forecasting, and performance analysis, all grounded in game-state and play-by-play data. This motivates leveraging tactical knowledge as a prior to support computer-vision-based predictions, enabling more automated and reliable extraction of play-by-play data. We introduce Footovision Play-by-Play Action Spotting in Soccer Dataset (FOOTPASS), the first benchmark for play-by-play action spotting over entire soccer matches in a multi-modal, multi-agent tactical context. It enables the development of methods for player-centric action spotting that exploit both outputs from computer-vision tasks (e.g., tracking, identification) and prior knowledge of soccer, including its tactical regularities over long time horizons, to generate reliable play-by-play data streams. These streams form an essential input for data-driven sports analytics.