# Video-Arena: A Scalable Agentic Video Benchmark for Spatiotemporal Capability

arXiv: https://arxiv.org/abs/2610.14158 · HTML: https://mlsanity.com/paper/2610.14158/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 26 of 26)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [Cambrian-S: Towards Spatial Supersensing in Video](https://mlsanity.com/paper/2511.04670/) (2025)
- [DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning](https://mlsanity.com/paper/2106.06135/) (2021)
- [MIRAGE: The Illusion of Visual Understanding](https://mlsanity.com/paper/2603.21687/) (2026)
- [Accelerating Self-Play Learning in Go](https://mlsanity.com/paper/1902.10565/) (2019)
- [An Open Source Data Contamination Report for Large Language Models](https://mlsanity.com/paper/2310.17589/) (2023)
- [Lost in Time: A New Temporal Benchmark for VideoLLMs](https://mlsanity.com/paper/2410.07752/) (2024)
- [GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents](https://mlsanity.com/paper/2604.07429/) (2026)
- [ShuttleSet: A Human-Annotated Stroke-Level Singles Dataset for Badminton Tactical Analysis](https://mlsanity.com/paper/2306.04948/) (2023)
- [Proact-VL: A Proactive VideoLLM for Real-Time AI Companions](https://mlsanity.com/paper/2603.03447/) (2026)
- [GameVerse: Can Vision-Language Models Learn from Video-based Reflection?](https://mlsanity.com/paper/2603.06656/) (2026)
- [Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization](https://mlsanity.com/paper/2505.00812/) (2025)
- [Towards Spatial Supersensing in the Wild](https://mlsanity.com/paper/2607.13681/) (2026)
- [Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse](https://mlsanity.com/paper/2605.09965/) (2026)
- [SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions](https://mlsanity.com/paper/2608.29607/) (2026)
- [EgoEsportsQA: An Egocentric Video Benchmark for Perception and Reasoning in Esports](https://mlsanity.com/paper/2604.12320/) (2026)
- [SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence](https://mlsanity.com/paper/2605.31529/) (2026)

## Abstract (arXiv)

Understanding and tracking a dynamic world are fundamental to visual intelligence. Existing video benchmarks provide limited insight into whether models can maintain a coherent understanding of evolving world state. Building such a benchmark from real-world videos is challenging because the underlying world state is partially observable, making annotations hard to obtain. To close this gap, we introduce Video-Arena, a benchmark that turns competitive games into scalable, verifiable tests of how well models understand evolving world states from videos. Specifically, inspired by how humans understand dynamic events, we evaluate spatiotemporal world state understanding and tracking through three task categories: perceiving critical events, interpreting state updates, and tracking the resulting states. Our pipeline can generate videos and aligned reference labels from game records at scale. To keep evaluation affordable and reproducible, we curate a golden set of 180 tasks across 9 games and run evaluations under two protocols: native visual understanding and tool-assisted agentic analysis. The best-performing systems achieve F1 scores of and under the two protocols, respectively. The result highlights that native models struggle to perceive events reliably, while tool-assisted systems improve event recovery but still struggle to interpret state updates and track the resulting states consistently over time. Our project page is in https://videoarena-bench.github.io