mlsanityData to 7 Oct 2026

SportD: How do VLMs physically strategize?

arXiv 2607.14616 ↗First posted 16 Jul 2026cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision-language models (VLMs) can describe a scene, but can they act well within one? We study whether VLMs can make sound strategic decisions, using soccer as an objective testbed with quantifiably-valued actions. We introduce SportD, a dataset and evaluation consisting of 1421 decision scenarios across professional men's and women's soccer games, where a VLM must decide what action to take next. Models on average select the optimal action around 27% of the time, less often than the professional players, and capture markedly less of the value at stake. Furthermore, they exhibit a clear preference for safer actions, favoring lower-variance, lower-value choices that also make less physical progress toward goal. Frontier VLMs are better at estimating whether an action will succeed, placing the highest-success-probability action among their top choices in 72-85% of cases. Yet VLMs systematically conflate likelihood with value, assigning higher value to actions that are more likely to succeed (ρ=+0.30 to +0.52), despite no such relationship in the ground truth (ρ=-0.08). Modifying the deliberation instructions to encourage risk-taking brings the frontier models closer to the players' skill levels. SportD opens a new direction for rigorously evaluating physical strategic decision-making in VLMs, showing that careful decomposition of their choices can reveal the mechanisms underlying systematic biases such as risk aversion.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

30 of 50 references.

References · 30

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  11. 2022A Generalist Agentcited by 802
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
18 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  3. 2024Genie: Generative Interactive Environmentscited by 706
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  6. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  7. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  8. 2022Visual Spatial Reasoningcited by 288
  9. 2018SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videoscited by 148
  10. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  11. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  12. 2020SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videoscited by 129
  13. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 129
  14. 2018Actions Speak Louder Than Goals: Valuing Player Actions in Soccercited by 90
  15. 2020A framework for the fine-grained evaluation of the instantaneous expected value of soccer possessionscited by 24
  16. 2024SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Modelscited by 24
  17. 2025SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sportscited by 17
  18. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GenTac: Generative Modeling and Forecasting of Soccer Tactics
  2. 2025Generative World Models of Tasks: LLM-Driven Hierarchical Scaffolding for Embodied Agents
  3. 2025Domain Adaptation of VLM for Soccer Video Understanding
  4. 2026Can Vision Language Models Judge Action Quality? An Empirical Evaluation
  5. 2025Do We Need Large VLMs for Spotting Soccer Actions?
  6. 2021Towards optimized actions in critical situations of soccer games with deep reinforcement learning
  7. 2024X-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Model
  8. 2026Monte Carlo Pass Search: Using Trajectory Generation for 3D Counterfactual Pass Evaluation in Football