# Sporthesia: Augmenting Sports Videos Using Natural Language

arXiv: https://arxiv.org/abs/2209.03434 · HTML: https://mlsanity.com/paper/2209.03434/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2022.

## Built on by (most cited first, 11 of 11)

- [SoccerNet Game State Reconstruction: End-to-End Athlete Tracking and Identification on a Minimap](https://mlsanity.com/paper/2404.11335/) (2024)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2410.08474/) (2024)
- [XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis](https://mlsanity.com/paper/2301.10385/) (2023)
- [SportQA: A Benchmark for Sports Understanding in Large Language Models](https://mlsanity.com/paper/2402.15862/) (2024)
- [Where Are We So Far? Understanding Data Storytelling Tools from the Perspective of Human-AI Collaboration](https://mlsanity.com/paper/2309.15723/) (2023)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Language and Multimodal Models in Sports: A Survey of Datasets and Applications](https://mlsanity.com/paper/2406.12252/) (2024)
- [Zero-Shot Video Translation via Token Warping](https://mlsanity.com/paper/2402.12099/) (2024)
- [A Generative AI System for Biomedical Data Discovery with Grammar-Based Visualizations](https://mlsanity.com/paper/2509.16454/) (2025)
- [YAC: Bridging Natural Language and Interactive Visual Exploration with Generative AI for Biomedical Data Discovery](https://mlsanity.com/paper/2509.19182/) (2025)

## Abstract (arXiv)

Augmented sports videos, which combine visualizations and video effects to present data in actual scenes, can communicate insights engagingly and thus have been increasingly popular for sports enthusiasts around the world. Yet, creating augmented sports videos remains a challenging task, requiring considerable time and video editing skills. On the other hand, sports insights are often communicated using natural language, such as in commentaries, oral presentations, and articles, but usually lack visual cues. Thus, this work aims to facilitate the creation of augmented sports videos by enabling analysts to directly create visualizations embedded in videos using insights expressed in natural language. To achieve this goal, we propose a three-step approach - 1) detecting visualizable entities in the text, 2) mapping these entities into visualizations, and 3) scheduling these visualizations to play with the video - and analyzed 155 sports video clips and the accompanying commentaries for accomplishing these steps. Informed by our analysis, we have designed and implemented Sporthesia, a proof-of-concept system that takes racket-based sports videos and textual commentaries as the input and outputs augmented videos. We demonstrate Sporthesia's applicability in two exemplar scenarios, i.e., authoring augmented sports videos using text and augmenting historical sports videos based on auditory comments. A technical evaluation shows that Sporthesia achieves high accuracy (F1-score of 0.9) in detecting visualizable entities in the text. An expert evaluation with eight sports analysts suggests high utility, effectiveness, and satisfaction with our language-driven authoring method and provides insights for future improvement and opportunities.