VLMs for Videogame Data Annotation
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Vision Language Models (VLMs) and Artificial Intelligence (AI) agents have revolutionized how engineers approach complex problems in real-world applications. Their adoption in video games is on the other hand limited by the extreme variability of the synthetic scenarios and their poor compliance with real-world physics. Here we investigate the use of VLMs for annotating video game frame sequences with reward signals, a task with several potential applications including, among others, conditioned training and offline reinforcement learning. We show that VLMs often struggle to answer basic questions on racing video games (although we observed a similar behavior on other game genres) and discuss countermeasures such as VLM output mixing and prompt optimization. We also show how input sequence length, resolution, and question batching affect the annotation quality and its token consumption.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 15 references that are in the map. First posted on arXiv 6 Aug 2026.
Builds on
10 of 15 references.
References · 10
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2022A Generalist Agentcited by 802
- 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
- 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
- 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
- 2026Qwen3.5-Omni Technical Reportcited by 146
- 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
- 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
- 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
- 2024Learning to Move Like Professional Counter-Strike Playerscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Training a Conditioned Video Game Agent on a VLM Annotated Dataset
- 2025Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning
- 2025VideoGameBench: Can Vision-Language Models complete popular video games?
- 2024Code as Reward: Empowering Reinforcement Learning with VLMs
- 2024ViSTa Dataset: Do vision-language models understand sequential tasks?
- 2025Cultivating Game Sense for Yourself: Making VLMs Gaming Experts
- 2026How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos
- 2026VLM-Guided Experience Replay