# RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees

arXiv: https://arxiv.org/abs/2604.15736 · HTML: https://mlsanity.com/paper/2604.15736/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos](https://mlsanity.com/paper/1804.04527/) (2018)
- [SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos](https://mlsanity.com/paper/2011.13367/) (2020)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent](https://mlsanity.com/paper/2411.02937/) (2024)
- [SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries](https://mlsanity.com/paper/2304.04565/) (2023)
- [TacticAI: an AI assistant for football tactics](https://mlsanity.com/paper/2310.10553/) (2023)
- [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025)
- [Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports](https://mlsanity.com/paper/2401.01505/) (2024)
- [VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Views](https://mlsanity.com/paper/2304.04617/) (2023)
- [Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark](https://mlsanity.com/paper/2504.14693/) (2025)
- [GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generation](https://mlsanity.com/paper/2303.14655/) (2023)
- [SportsPose -- A Dynamic 3D sports pose dataset](https://mlsanity.com/paper/2304.01865/) (2023)
- [Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?](https://mlsanity.com/paper/2505.14321/) (2025)
- [TimeSoccer: An End-to-End Multimodal Large Language Model for Soccer Commentary Generation](https://mlsanity.com/paper/2504.17365/) (2025)
- [Language and Multimodal Models in Sports: A Survey of Datasets and Applications](https://mlsanity.com/paper/2406.12252/) (2024)
- [FineBadminton: A Multi-Level Dataset for Fine-Grained Badminton Video Understanding](https://mlsanity.com/paper/2508.07554/) (2025)
- [SportsSum2.0: Generating High-Quality Sports News from Live Text Commentary](https://mlsanity.com/paper/2110.05750/) (2021)
- [Enhancing Soccer Camera Calibration Through Keypoint Exploitation](https://mlsanity.com/paper/2410.07401/) (2024)
- [SCBench: A Sports Commentary Benchmark for Video LLMs](https://mlsanity.com/paper/2412.17637/) (2024)
- [FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding](https://mlsanity.com/paper/2504.19514/) (2025)
- [SoccerHigh: A Benchmark Dataset for Automatic Soccer Video Summarization](https://mlsanity.com/paper/2509.01439/) (2025)
- [SportsGPT: An LLM-driven Framework for Interpretable Sports Motion Assessment and Training Guidance](https://mlsanity.com/paper/2512.14121/) (2025)

## Built on by (most cited first, 1 of 1)

- [Prompt, Probe, Train, or Annotate? Single-camera sports video understanding in amateur settings](https://mlsanity.com/paper/2609.28049/) (2026)

## Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) excel at generic video understanding, their ability to support specialized, rule-grounded decision-making remains insufficiently explored. In this paper, we introduce RefereeBench, the first large-scale benchmark for evaluating MLLMs as automatic sports referees. Spanning 11 sports with 925 curated videos and 6,475 QA pairs, RefereeBench evaluates five core officiating abilities: foul existence, foul and penalty classification, foul and penalty reasoning, entity perception, and temporal grounding. The benchmark is fully human-annotated to ensure high-quality annotations grounded in authentic officiating logic and multimodal evidence. Extensive evaluations of state-of-the-art MLLMs show that even the strongest models, such as Doubao-Seed-1.8 and Gemini-3-Pro, achieve only around 60% accuracy, while the strongest open-source model, Qwen3-VL, reaches only 47%. These results indicate that current models remain far from being reliable sports referees. Further analysis shows that while models can often identify incidents and involved entities, they struggle with rule application and temporal grounding, and frequently over-call fouls on normal clips. Our benchmark highlights the need for future MLLMs that better integrate domain knowledge and multimodal understanding, advancing trustworthy AI-assisted officiating and broader multimodal decision-making.