# SCBench: A Sports Commentary Benchmark for Video LLMs

arXiv: https://arxiv.org/abs/2412.17637 · HTML: https://mlsanity.com/paper/2412.17637/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2403.16999/) (2024)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input](https://mlsanity.com/paper/2408.15542/) (2024)
- [SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos](https://mlsanity.com/paper/2011.13367/) (2020)
- [Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models](https://mlsanity.com/paper/2311.16103/) (2023)
- [Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance](https://mlsanity.com/paper/2410.16261/) (2024)
- [ChatPose: Chatting about 3D Human Pose](https://mlsanity.com/paper/2311.18836/) (2023)
- [FineAction: A Fine-Grained Video Dataset for Temporal Action Localization](https://mlsanity.com/paper/2105.11107/) (2021)
- [AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering](https://mlsanity.com/paper/2311.14906/) (2023)
- [A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming](https://mlsanity.com/paper/2404.16038/) (2024)
- [A Survey of Video Datasets for Grounded Event Understanding](https://mlsanity.com/paper/2406.09646/) (2024)
- [Cross-Block Fine-Grained Semantic Cascade for Skeleton-Based Sports Action Recognition](https://mlsanity.com/paper/2404.19383/) (2024)

## Built on by (most cited first, 7 of 7)

- [SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding](https://mlsanity.com/paper/2505.16630/) (2025)
- [Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches](https://mlsanity.com/paper/2603.02655/) (2026)
- [TennisExpert: Towards Expert-Level Analytical Sports Video Understanding](https://mlsanity.com/paper/2603.13397/) (2026)
- [RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees](https://mlsanity.com/paper/2604.15736/) (2026)
- [Agentic generative AI for media content discovery at the national football league](https://mlsanity.com/paper/2510.07297/) (2025)
- [BoxComm: Benchmarking Category-Aware Commentary Generation and Narration Rhythm in Boxing](https://mlsanity.com/paper/2604.04419/) (2026)
- [A Survey of Large Models in Sports](https://mlsanity.com/paper/2608.14377/) (2026)

## Abstract (arXiv)

Recently, significant advances have been made in Video Large Language Models (Video LLMs) in both academia and industry. However, methods to evaluate and benchmark the performance of different Video LLMs, especially their fine-grained, temporal visual capabilities, remain very limited. On one hand, current benchmarks use relatively simple videos (e.g., subtitled movie clips) where the model can understand the entire video by processing just a few frames. On the other hand, their datasets lack diversity in task format, comprising only QA or multi-choice QA, which overlooks the models' capacity for generating in-depth and precise texts. Sports videos, which feature intricate visual information, sequential events, and emotionally charged commentary, present a critical challenge for Video LLMs, making sports commentary an ideal benchmarking task. Inspired by these challenges, we propose a novel task: sports video commentary generation, developed SCBench for Video LLMs. To construct such a benchmark, we introduce (1) SCORES, a six-dimensional metric specifically designed for our task, upon which we propose a GPT-based evaluation method, and (2) CommentarySet, a dataset consisting of 5,775 annotated video clips and ground-truth labels tailored to our metric. Based on SCBench, we conduct comprehensive evaluations on multiple Video LLMs (e.g. VILA, Video-LLaVA, etc.) and chain-of-thought baseline methods. Our results found that InternVL-Chat-2 achieves the best performance with 5.44, surpassing the second-best by 1.04. Our work provides a fresh perspective for future research, aiming to enhance models' overall capabilities in complex visual understanding tasks. Our dataset will be released soon.