# Language and Multimodal Models in Sports: A Survey of Datasets and Applications

arXiv: https://arxiv.org/abs/2406.12252 · HTML: https://mlsanity.com/paper/2406.12252/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 46 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [Large-Scale Hate Speech Detection with Cross-Domain Transfer](https://mlsanity.com/paper/2203.01111/) (2022)
- [Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports](https://mlsanity.com/paper/2401.01505/) (2024)
- [Hate Speech detection in the Bengali language: A dataset and its baseline evaluation](https://mlsanity.com/paper/2012.09686/) (2020)
- [GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generation](https://mlsanity.com/paper/2303.14655/) (2023)
- [LiveQA: A Question Answering Dataset over Sports Live](https://mlsanity.com/paper/2010.00526/) (2020)
- [FSD-10: A Dataset for Competitive Sports Content Analysis](https://mlsanity.com/paper/2002.03312/) (2020)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [Sporthesia: Augmenting Sports Videos Using Natural Language](https://mlsanity.com/paper/2209.03434/) (2022)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [SportsSum2.0: Generating High-Quality Sports News from Live Text Commentary](https://mlsanity.com/paper/2110.05750/) (2021)
- [VREN: Volleyball Rally Dataset with Expression Notation Language](https://mlsanity.com/paper/2209.13846/) (2022)
- [A Unified Taxonomy and Multimodal Dataset for Events in Invasion Games](https://mlsanity.com/paper/2108.11149/) (2021)
- [Evaluating and Enhancing Large Language Models Performance in Domain-specific Medicine: Osteoarthritis Management with DocOA](https://mlsanity.com/paper/2401.12998/) (2024)

## Built on by (most cited first, 10 of 10)

- [SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2410.08474/) (2024)
- [SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports](https://mlsanity.com/paper/2511.06499/) (2025)
- [MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs](https://mlsanity.com/paper/2508.10264/) (2025)
- [SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding](https://mlsanity.com/paper/2505.16630/) (2025)
- [Sports Intelligence: Assessing the Sports Understanding Capabilities of Language Models through Question Answering from Text to Video](https://mlsanity.com/paper/2406.14877/) (2024)
- [DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning](https://mlsanity.com/paper/2511.12908/) (2025)
- [RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees](https://mlsanity.com/paper/2604.15736/) (2026)
- [From Evidence-Based Medicine to Knowledge Graph: Retrieval-Augmented Generation for Sports Rehabilitation and a Domain Benchmark](https://mlsanity.com/paper/2601.00216/) (2026)
- [BoxComm: Benchmarking Category-Aware Commentary Generation and Narration Rhythm in Boxing](https://mlsanity.com/paper/2604.04419/) (2026)
- [A Survey of Large Models in Sports](https://mlsanity.com/paper/2608.14377/) (2026)

## Abstract (arXiv)

Recent integration of Natural Language Processing (NLP) and multimodal models has advanced the field of sports analytics. This survey presents a comprehensive review of the datasets and applications driving these innovations post-2020. We overviewed and categorized datasets into three primary types: language-based, multimodal, and convertible datasets. Language-based and multimodal datasets are for tasks involving text or multimodality (e.g., text, video, audio), respectively. Convertible datasets, initially single-modal (video), can be enriched with additional annotations, such as explanations of actions and video descriptions, to become multimodal, offering future potential for richer and more diverse applications. Our study highlights the contributions of these datasets to various applications, from improving fan experiences to supporting tactical analysis and medical diagnostics. We also discuss the challenges and future directions in dataset development, emphasizing the need for diverse, high-quality data to support real-time processing and personalized user experiences. This survey provides a foundational resource for researchers and practitioners aiming to leverage NLP and multimodal models in sports, offering insights into current trends and future opportunities in the field.