mlsanityData to 7 Oct 2026

See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay

arXiv 2603.11601 ↗First posted 12 Mar 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions. We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments. We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines. Our results indicate that all models benefit when the symbolic information is accurate. However, when VLMs extract symbols themselves, performance becomes dependent on model capability and scene complexity. We further investigate how accurately VLMs can extract symbolic information from visual inputs and how noise in these symbols affects decision-making and gameplay performance. Our findings reveal that symbolic grounding is beneficial in VLMs only when symbol extraction is reliable, and highlight perception quality as a central bottleneck for future VLM-based agents.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 5 of its 19 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

5 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Games28 Aug

Other references · 4

  1. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  2. 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 25
  3. 2020Planning from Pixels in Atari with Learned Symbolic Representationscited by 11
  4. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoGameBench: Can Vision-Language Models complete popular video games?
  2. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments
  3. 2026How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos
  4. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models
  5. 2026GameVerse: Can Vision-Language Models Learn from Video-based Reflection?
  6. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective
  7. 2026VLMs for Videogame Data Annotation
  8. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization