# Multi-Modal Soccer Scene Analysis with Masked Pre-Training

arXiv: https://arxiv.org/abs/2512.19528 · HTML: https://mlsanity.com/paper/2512.19528/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 22 Dec 2025.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [BRITS: Bidirectional Recurrent Imputation for Time Series](https://mlsanity.com/paper/1805.10572/) (2018)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes](https://mlsanity.com/paper/2304.05170/) (2023)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark in Soccer Videos](https://mlsanity.com/paper/2204.06918/) (2022)
- [NAOMI: Non-Autoregressive Multiresolution Sequence Imputation](https://mlsanity.com/paper/1901.10946/) (2019)
- [Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction](https://mlsanity.com/paper/2303.16005/) (2023)
- [Generating Multi-Agent Trajectories using Programmatic Weak Supervision](https://mlsanity.com/paper/1803.07612/) (2018)
- [TacticAI: an AI assistant for football tactics](https://mlsanity.com/paper/2310.10553/) (2023)
- [Camera Calibration and Player Localization in SoccerNet-v2 and Investigation of their Representations for Action Spotting](https://mlsanity.com/paper/2104.09333/) (2021)
- [MatchTime: Towards Automatic Soccer Game Commentary Generation](https://mlsanity.com/paper/2406.18530/) (2024)
- [SoccerNet Game State Reconstruction: End-to-End Athlete Tracking and Identification on a Minimap](https://mlsanity.com/paper/2404.11335/) (2024)
- [Generating Long-term Trajectories Using Deep Hierarchical Networks](https://mlsanity.com/paper/1706.07138/) (2017)
- [Towards Universal Soccer Video Understanding](https://mlsanity.com/paper/2412.01820/) (2024)
- [Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports](https://mlsanity.com/paper/2405.17680/) (2024)
- [Multi-Agent System for Comprehensive Soccer Understanding](https://mlsanity.com/paper/2505.03735/) (2025)
- [Group Activity Detection from Trajectory and Video Data in Soccer](https://mlsanity.com/paper/2004.10299/) (2020)
- [Learning Better Masking for Better Language Model Pre-training](https://mlsanity.com/paper/2208.10806/) (2022)
- [Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2503.18589/) (2025)
- [Towards Active Learning for Action Spotting in Association Football Videos](https://mlsanity.com/paper/2304.04220/) (2023)
- [PnLCalib: Sports Field Registration via Points and Lines Optimization](https://mlsanity.com/paper/2404.08401/) (2024)

## Built on by (most cited first, 1 of 1)

- [PathCRF: Ball-Free Soccer Event Detection via Possession Path Inference from Player Trajectories](https://mlsanity.com/paper/2602.12080/) (2026)

## Abstract (arXiv)

In this work we propose a multi-modal architecture for analyzing soccer scenes from tactical camera footage, with a focus on three core tasks: ball trajectory inference, ball state classification, and ball possessor identification. To this end, our solution integrates three distinct input modalities (player trajectories, player types and image crops of individual players) into a unified framework that processes spatial and temporal dynamics using a cascade of sociotemporal transformer blocks. Unlike prior methods, which rely heavily on accurate ball tracking or handcrafted heuristics, our approach infers the ball trajectory without direct access to its past or future positions, and robustly identifies the ball state and ball possessor under noisy or occluded conditions from real top league matches. We also introduce CropDrop, a modality-specific masking pre-training strategy that prevents over-reliance on image features and encourages the model to rely on cross-modal patterns during pre-training. We show the effectiveness of our approach on a large-scale dataset providing substantial improvements over state-of-the-art baselines in all tasks. Our results highlight the benefits of combining structured and visual cues in a transformer-based architecture, and the importance of realistic masking strategies in multi-modal learning.