# VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model arXiv: https://arxiv.org/abs/2502.18906 ยท HTML: https://mlsanity.com/paper/2502.18906/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. - Builds on 54 of its 64 references that are in the map. - First posted on arXiv 26 Feb 2025. ## Placed in - LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/) - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) - LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/) ## Builds on (most cited first, 30 of 54) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020) - [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021) - [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020) - [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024) - [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024) - [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023) - [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023) - [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023) - [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023) - [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024) - [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024) - [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023) - [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024) - [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024) - [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023) - [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024) - [Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning](https://mlsanity.com/paper/2410.08146/) (2024) - [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024) - [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024) - [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022) - [DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning](https://mlsanity.com/paper/2406.11896/) (2024) - [Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning](https://mlsanity.com/paper/2305.14909/) (2023) - [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023) - [You Only Look at Screens: Multimodal Chain-of-Action Agents](https://mlsanity.com/paper/2309.11436/) (2023) - [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024) ## Built on by (most cited first, 19 of 19) - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025) - [GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents](https://mlsanity.com/paper/2506.03143/) (2025) - [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025) - [API Agents vs. GUI Agents: Divergence and Convergence](https://mlsanity.com/paper/2503.11069/) (2025) - [Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation](https://mlsanity.com/paper/2506.04614/) (2025) - [GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks](https://mlsanity.com/paper/2509.23738/) (2025) - [ProgRM: Build Better GUI Agents with Progress Rewards](https://mlsanity.com/paper/2505.18121/) (2025) - [a1: Steep Test-time Scaling Law via Environment Augmented Generation](https://mlsanity.com/paper/2504.14597/) (2025) - [BIMgent: Towards Autonomous Building Modeling via Computer-use Agents](https://mlsanity.com/paper/2506.07217/) (2025) - [GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents](https://mlsanity.com/paper/2511.04307/) (2025) - [Large Language Models as Computable Approximations to Solomonoff Induction](https://mlsanity.com/paper/2505.15784/) (2025) - [UFO3: Weaving the Digital Agent Galaxy](https://mlsanity.com/paper/2511.11332/) (2025) - [Gated Differentiable Working Memory for Long-Context Language Modeling](https://mlsanity.com/paper/2601.12906/) (2026) - [Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions](https://mlsanity.com/paper/2604.07277/) (2026) - [BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent](https://mlsanity.com/paper/2509.15566/) (2025) - [InfraMind: A Novel Exploration-based GUI Agentic Framework for Mission-critical Industrial Management](https://mlsanity.com/paper/2509.13704/) (2025) - [GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models](https://mlsanity.com/paper/2601.18197/) (2026) - [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026) - [AutoGUIWorld: Image Generators as Visual World Models for GUI Agent](https://mlsanity.com/paper/2610.01215/) (2026) ## Abstract (arXiv) Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI) agents via Reinforcement Learning (RL) faces critical challenges: environment-based RL requires costly interactions, while environment-free methods struggle with distribution shift and reward generalization. We propose an environment-free RL framework that decouples value estimation from policy optimization by leveraging a pretrained Value Environment Model (VEM). VEM predicts state-action values directly from offline data, distilling human-like priors about GUI interaction outcomes without requiring next-state prediction or environmental feedback. This avoids compounding errors and enhances resilience to UI changes by focusing on semantic reasoning (e.g., Does this action advance the user's goal?). The framework operates in two stages: (1) pretraining VEM to estimate long-term action utilities and (2) guiding policy exploration with frozen VEM signals, enabling layout-agnostic GUI automation. Evaluated on Android-in-the-Wild benchmarks, VEM achieves state-of-the-art performance in both offline and online settings, outperforming environment-free baselines significantly and matching environment-based approaches without interaction costs. Importantly, VEM demonstrates that semantic-aware value estimation can achieve comparable performance with online-trained methods.