mlsanityData to 7 Oct 2026

X-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Model

arXiv 2404.06332 ↗First posted 7 Apr 2024cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The rapid advancement of artificial intelligence has led to significant improvements in automated decision-making. However, the increased performance of models often comes at the cost of explainability and transparency of their decision-making processes. In this paper, we investigate the capabilities of large language models to explain decisions, using football refereeing as a testing ground, given its decision complexity and subjectivity. We introduce the Explainable Video Assistant Referee System, X-VARS, a multi-modal large language model designed for understanding football videos from the point of view of a referee. X-VARS can perform a multitude of tasks, including video description, question answering, action recognition, and conducting meaningful conversations based on video content and in accordance with the Laws of the Game for football referees. We validate X-VARS on our novel dataset, SoccerNet-XFoul, which consists of more than 22k video-question-answer triplets annotated by over 70 experienced football referees. Our experiments and human study illustrate the impressive capabilities of X-VARS in interpreting complex football clips. Furthermore, we highlight the potential of X-VARS to reach human performance and support football referees in the future.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 80 references that are in the map. First posted on arXiv 7 Apr 2024.

Builds on

58 of 80 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2021Multiscale Vision Transformerscited by 896
  12. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  13. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  14. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  15. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  16. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  17. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  18. 2018SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videoscited by 148
  19. 2020SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videoscited by 129
  20. 2022SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark in Soccer Videoscited by 80
  21. 2022Body Part-Based Representation Learning for Occluded Person Re-Identificationcited by 63
  22. 2023SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentariescited by 57
  23. 2019A Context-Aware Loss Function for Action Spotting in Soccer Videoscited by 56
  24. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  25. 2022Rethinking Explainability as a Dialogue: A Practitioner's Perspectivecited by 52
  26. 2021Camera Calibration and Player Localization in SoccerNet-v2 and Investigation of their Representations for Action Spottingcited by 43
  27. 2024Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sportscited by 41
  28. 2022Spotting Temporally Precise, Fine-Grained Events in Videocited by 40
  29. 2023VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Viewscited by 38
  30. 2022DeepSportradar-v1: Computer Vision Dataset for Sports Understanding with High Quality Annotationscited by 35
  31. 2022SoccerNet 2022 Challenges Resultscited by 28
  32. 2022Temporally Precise Action Spotting in Soccer Videos Using Dense Detection Anchorscited by 27
  33. 2022Semi-Supervised Training to Improve Player and Ball Detection in Soccercited by 25
  34. 2020Using Player's Body-Orientation to Model Pass Feasibility in Soccercited by 20
  35. 2023SoccerNet 2023 Challenges Resultscited by 20
  36. 2022Efficient tracking of team sport players with few game-specific annotationscited by 17
  37. 2023COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformerscited by 17
  38. 2022Action Spotting using Dense Detection Anchors Revisited: Submission to the SoccerNet Challenge 2022cited by 14

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 7; 2026: 6.

22 papers.

Sort:

By area:

  1. 2024SoccerNet Game State Reconstruction: End-to-End Athlete Tracking and Identification on a Minimapcited by 34
  2. 2024SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Modelscited by 24
  3. 2024Towards Universal Soccer Video Understandingcited by 21
  4. 2024A Universal Protocol to Benchmark Camera Calibration for Sportscited by 14
  5. 2024SoccerNet-Echoes: A Soccer Game Audio Commentary Datasetcited by 12
  6. 2025SoccerMaster: A Vision Foundation Model for Soccer Understandingcited by 10
  7. 2025Action Anticipation from SoccerNet Football Video Broadcastscited by 9
  8. 2024Towards AI-Powered Video Assistant Referee System (VARS) for Association Footballcited by 8
  9. 2024SoccerNet 2024 Challenges Resultscited by 8
  10. 2025SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understandingcited by 8
  11. 2024Deep learning for action spotting in association football videoscited by 7
  12. 2025SoccerNet 2025 Challenges Resultscited by 4
10 more
  1. 2025SV3.3B: A Sports Video Understanding Model for Action Recognitioncited by 3
  2. 2024OSL-ActionSpotting: A Unified Library for Action Spotting in Sports Videoscited by 2
  3. 2025From Broadcast to Minimap: Achieving State-of-the-Art SoccerNet Game State Reconstructioncited by 1
  4. 2026Towards Athlete Fatigue Assessment from Association Football Videoscited by 1
  5. 2026SoccerNet 2026 Challenges Resultscited by 1
  6. 2025Agentic generative AI for media content discovery at the national football leaguecited by 0
  7. 2026BFMD: A Full-Match Badminton Dense Dataset for Dense Shot Captioningcited by 0
  8. 2026SoccerRef-Agents: Multi-Agent System for Automated Soccer Refereeingcited by 0
  9. 2026SoccerLens: Grounded Soccer Video Understanding Beyond Accuracycited by 0
  10. 2026TreeSoc: Tree-Structured Dynamic Reasoning and Tool Synergy for Soccer Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees
  2. 2024Sports Intelligence: Assessing the Sports Understanding Capabilities of Language Models through Question Answering from Text to Video
  3. 2025SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports
  4. 2025Multi-Agent System for Comprehensive Soccer Understanding
  5. 2026Beyond the Single Camera: Agentic Multi-View Reasoning in Sports Video Understanding
  6. 2025Do We Need Large VLMs for Spotting Soccer Actions?
  7. 2026MSUE: Multi-Modal Soccer Understanding Expert
  8. 2025TennisTV: Do Multimodal Large Language Models Understand Tennis Rallies?