# What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

arXiv: https://arxiv.org/abs/2605.03782 · HTML: https://mlsanity.com/paper/2605.03782/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 20 of its 54 references that are in the map.
- First posted on arXiv 5 May 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI](https://mlsanity.com/paper/2410.00425/) (2024)
- [EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents](https://mlsanity.com/paper/2502.09560/) (2025)
- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models](https://mlsanity.com/paper/1507.00814/) (2015)
- [Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents](https://mlsanity.com/paper/2411.06559/) (2024)
- [VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents](https://mlsanity.com/paper/2510.16907/) (2025)
- [Gymnasium: A Standard Interface for Reinforcement Learning Environments](https://mlsanity.com/paper/2407.17032/) (2024)
- [Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective](https://mlsanity.com/paper/2510.15047/) (2025)
- [Critique of World Model](https://mlsanity.com/paper/2507.05169/) (2025)
- [Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration](https://mlsanity.com/paper/2109.08603/) (2021)
- [World Discovery Models](https://mlsanity.com/paper/1902.07685/) (2019)
- [Active Intelligence in Video Avatars via Closed-loop World Modeling](https://mlsanity.com/paper/2512.20615/) (2025)
- [Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction](https://mlsanity.com/paper/2511.23476/) (2025)

## Built on by (most cited first, 1 of 1)

- [PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents](https://mlsanity.com/paper/2609.35303/) (2026)

## Abstract (arXiv)

To navigate partially observable visual environments, recent VLM agents increasingly internalize world modeling capabilities into their policies via explicit CoT reasoning, enabling them to mentally simulate futures before acting. However, relying solely on passive reasoning over visited states is insufficient for sparse-reward tasks, as it lacks the epistemic drive to actively uncover the ``known unknown'' required for robust generalization. We ask: Can VLM agents actively find signals that challenge and refine their internal world model through curiosity-driven exploration? In this work, we propose GLANCE, a unified framework that bridges reasoning and exploration by grounding the agent's linguistic world model into the stable visual representations of an evolving target network. Crucially, GLANCE leverages the discrepancy between linguistic prediction and visual reality as an intrinsic curiosity signal within reinforcement learning, steering the agent to actively explore areas where its internal model is uncertain. Extensive experiments across a series of agentic tasks show the effectiveness of GLANCE, and demonstrate that aligning ``what the agent thinks'' with ``what the agent sees'' is key to solving complex or sparse agentic tasks.