# WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model

arXiv: https://arxiv.org/abs/2504.21024 · HTML: https://mlsanity.com/paper/2504.21024/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 42 of its 54 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 30 of 42)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents](https://mlsanity.com/paper/2408.07199/) (2024)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents](https://mlsanity.com/paper/2411.06559/) (2024)
- [Tree Search for Language Model Agents](https://mlsanity.com/paper/2407.01476/) (2024)
- [Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation](https://mlsanity.com/paper/2410.13232/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [AgentGym: Evolving Large Language Model-based Agents across Diverse Environments](https://mlsanity.com/paper/2406.04151/) (2024)
- [WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration](https://mlsanity.com/paper/2408.15978/) (2024)
- [ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent](https://mlsanity.com/paper/2312.10003/) (2023)
- [Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents](https://mlsanity.com/paper/2502.11357/) (2025)
- [Agent Planning with World Knowledge Model](https://mlsanity.com/paper/2405.14205/) (2024)
- [ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning](https://mlsanity.com/paper/2410.02052/) (2024)
- [NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild](https://mlsanity.com/paper/2410.02907/) (2024)

## Built on by (most cited first, 30 of 61)

- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [SkillClaw: Let Skills Evolve Collectively with Agentic Evolver](https://mlsanity.com/paper/2604.08377/) (2026)
- [MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment](https://mlsanity.com/paper/2507.05720/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Scaling Agent Learning via Experience Synthesis](https://mlsanity.com/paper/2511.03773/) (2025)
- [Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training](https://mlsanity.com/paper/2508.00414/) (2025)
- [Agentic Entropy-Balanced Policy Optimization](https://mlsanity.com/paper/2510.14545/) (2025)
- [WebWorld: A Large-Scale World Model for Web Agent Training](https://mlsanity.com/paper/2602.14721/) (2026)
- [K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model](https://mlsanity.com/paper/2602.19128/) (2026)
- [Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification](https://mlsanity.com/paper/2601.15808/) (2026)
- [WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis](https://mlsanity.com/paper/2507.04370/) (2025)
- [DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking](https://mlsanity.com/paper/2510.20168/) (2025)
- [Test-Time Adaptation for LLM Agents via Environment Interaction](https://mlsanity.com/paper/2511.04847/) (2025)
- [Recursive Synthesis for Long-Horizon Terminal Tasks](https://mlsanity.com/paper/2608.05466/) (2026)
- [LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training](https://mlsanity.com/paper/2510.14969/) (2025)
- [WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback](https://mlsanity.com/paper/2505.20013/) (2025)
- [SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning](https://mlsanity.com/paper/2605.09423/) (2026)
- [WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance](https://mlsanity.com/paper/2511.12997/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Policy and World Modeling Co-Training for Language Agents](https://mlsanity.com/paper/2606.02388/) (2026)
- [Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agents](https://mlsanity.com/paper/2506.00320/) (2025)
- [DynaWeb: Model-Based Reinforcement Learning of Web Agents](https://mlsanity.com/paper/2601.22149/) (2026)
- [Generative Visual Code Mobile World Models](https://mlsanity.com/paper/2602.01576/) (2026)
- [Self-Evolving World Models for LLM Agent Planning](https://mlsanity.com/paper/2606.30639/) (2026)
- [R-WoM: Retrieval-augmented World Model For Computer-use Agents](https://mlsanity.com/paper/2510.11892/) (2025)
- [SafePred: A Predictive Guardrail for Computer-Using Agents via World Models](https://mlsanity.com/paper/2602.01725/) (2026)
- [World-Model-Augmented Web Agents with Action Correction](https://mlsanity.com/paper/2602.15384/) (2026)
- [SEVerA: Verified Synthesis of Self-Evolving Agents](https://mlsanity.com/paper/2603.25111/) (2026)
- [Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis](https://mlsanity.com/paper/2605.14392/) (2026)
- [OSExpert: Computer-Use Agents Learning Professional Skills via Exploration](https://mlsanity.com/paper/2603.07978/) (2026)

## Abstract (arXiv)

Agent self-improvement, where the backbone Large Language Model (LLM) of the agent are trained on trajectories sampled autonomously based on their own policies, has emerged as a promising approach for enhancing performance. Recent advancements, particularly in web environments, face a critical limitation: their performance will reach a stagnation point during autonomous learning cycles, hindering further improvement. We argue that this stems from limited exploration of the web environment and insufficient exploitation of pre-trained web knowledge in LLMs. To improve the performance of self-improvement, we propose a novel framework that introduces a co-evolving World Model LLM. This world model predicts the next observation based on the current observation and action within the web environment. Leveraging LLMs' pretrained knowledge of abundant web content, the World Model serves dual roles: (1) as a virtual web server generating self-instructed training data to continuously refine the agent's policy, and (2) as an imagination engine during inference, enabling look-ahead simulation to guide action selection for the agent LLM. Experiments in real-world web environments (Mind2Web-Live, WebVoyager, and GAIA-web) show a 10% performance gain over existing self-evolving agents, demonstrating the efficacy and generalizability of our approach, without using any distillation from more powerful close-sourced models. Our work establishes the necessity of integrating world models into autonomous agent frameworks to unlock sustained adaptability. Code is available at https://github.com/Tencent/SelfEvolvingAgent