mlsanityData to 14 Oct 2026

Training a Conditioned Video Game Agent on a VLM Annotated Dataset

arXiv 2608.05954 ↗First posted 6 Aug 2026cs.AI cs.CV cs.LGcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement Learning (RL) is a powerful but far from easy-to-use technique for policy learning. In the specific case of video games, access to the game engine is required to get rewards for training (e.g. to collect rewards from the environment). Furthermore, the proper identification and weighting of the rewards generally requires a difficult trial-and-error approach. Lastly, rewards are often sparse and understanding how they eventually affect the learned policy is a non-trivial exercise. To ease these issues we propose annotating a video game dataset with Vision Language Models (VLMs) instructed to extract human defined rewards. We show that offline RL can then be used to train a conditioned agent that responds accordingly to the desired returns and we discuss the difficulties and limitations that emerged in our early experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 7 of its 13 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

7 of 13 references.

References · 7

  1. 2016Generative Adversarial Imitation Learningcited by 1,597
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  5. 2022Multi-Game Decision Transformerscited by 214
  6. 2026Qwen3.5-Omni Technical Reportcited by 174
  7. 2024Learning to Move Like Professional Counter-Strike Playerscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  3. 2026VLMs for Videogame Data Annotation
  4. 2025Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning
  5. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
  6. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning
  7. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning
  8. 2023Vision-Language Models as a Source of Rewards