# SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2410.08474 · HTML: https://mlsanity.com/paper/2410.08474/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 49 of its 68 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 49)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [Tarsier: Recipes for Training and Evaluating Large Video Description Models](https://mlsanity.com/paper/2407.00634/) (2024)
- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://mlsanity.com/paper/2109.06379/) (2021)
- [MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos](https://mlsanity.com/paper/2406.08407/) (2024)
- [Spatio-Temporal Dynamic Inference Network for Group Activity Recognition](https://mlsanity.com/paper/2108.11743/) (2021)
- [Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports](https://mlsanity.com/paper/2401.01505/) (2024)
- [VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Views](https://mlsanity.com/paper/2304.04617/) (2023)
- [Semi-Supervised Training to Improve Player and Ball Detection in Soccer](https://mlsanity.com/paper/2204.06859/) (2022)
- [FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment](https://mlsanity.com/paper/2405.06887/) (2024)

## Built on by (most cited first, 24 of 24)

- [OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM](https://mlsanity.com/paper/2510.15870/) (2025)
- [FaceXBench: Evaluating Multimodal LLMs on Face Understanding](https://mlsanity.com/paper/2501.10360/) (2025)
- [SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence](https://mlsanity.com/paper/2505.17012/) (2025)
- [Towards Universal Soccer Video Understanding](https://mlsanity.com/paper/2412.01820/) (2024)
- [SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports](https://mlsanity.com/paper/2511.06499/) (2025)
- [SoccerMaster: A Vision Foundation Model for Soccer Understanding](https://mlsanity.com/paper/2512.11016/) (2025)
- [SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding](https://mlsanity.com/paper/2505.16630/) (2025)
- [VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding](https://mlsanity.com/paper/2602.04094/) (2026)
- [FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding](https://mlsanity.com/paper/2504.19514/) (2025)
- [FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoning](https://mlsanity.com/paper/2509.11796/) (2025)
- [DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning](https://mlsanity.com/paper/2511.12908/) (2025)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [SV3.3B: A Sports Video Understanding Model for Action Recognition](https://mlsanity.com/paper/2507.17844/) (2025)
- [TennisTV: Do Multimodal Large Language Models Understand Tennis Rallies?](https://mlsanity.com/paper/2509.15602/) (2025)
- [Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports](https://mlsanity.com/paper/2603.09896/) (2026)
- [WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting](https://mlsanity.com/paper/2607.18084/) (2026)
- [Learning Consistent Temporal Grounding between Related Tasks in Sports Coaching](https://mlsanity.com/paper/2603.18453/) (2026)
- [Towards Temporal Compositional Reasoning in Long-Form Sports Videos](https://mlsanity.com/paper/2604.22226/) (2026)
- [SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence](https://mlsanity.com/paper/2605.31529/) (2026)
- [Beyond the Single Camera: Agentic Multi-View Reasoning in Sports Video Understanding](https://mlsanity.com/paper/2607.11844/) (2026)
- [Agentic generative AI for media content discovery at the national football league](https://mlsanity.com/paper/2510.07297/) (2025)
- [SoccerRef-Agents: Multi-Agent System for Automated Soccer Refereeing](https://mlsanity.com/paper/2604.23392/) (2026)
- [Counterfactual Reasoning for Fine-Grained Evidence Disentanglement in VideoQA](https://mlsanity.com/paper/2606.09181/) (2026)
- [SportD: How do VLMs physically strategize?](https://mlsanity.com/paper/2607.14616/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are advancing the ability to reason about complex sports scenarios by integrating textual and visual information. To comprehensively evaluate their capabilities, we introduce SPORTU, a benchmark designed to assess MLLMs across multi-level sports reasoning tasks. SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models' ability to reason about sports solely through question-answering (QA), without requiring visual inputs; SPORTU-video, consisting of 1,701 slow-motion video clips across 7 different sports and 12,048 QA pairs, designed to assess multi-level reasoning, from simple sports recognition to complex tasks like foul detection and rule application. We evaluate four prevalent LLMs mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting on the SPORTU-text part. We evaluate four LLMs using few-shot learning and chain-of-thought (CoT) prompting on SPORTU-text. GPT-4o achieves the highest accuracy of 71%, but still falls short of human-level performance, highlighting room for improvement in rule comprehension and reasoning. The evaluation for the SPORTU-video part includes 7 proprietary and 6 open-source MLLMs. Experiments show that models fall short on hard tasks that require deep reasoning and rule-based understanding. Claude-3.5-Sonnet performs the best with only 52.6% accuracy on the hard task, showing large room for improvement. We hope that SPORTU will serve as a critical step toward evaluating models' capabilities in sports understanding and reasoning.