mlsanityData to 14 Oct 2026

WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model

arXiv 2504.21024 ↗First posted 23 Apr 2025cs.CLcited by 61

On the map: World models → World models for language agents

also in 2 more
Abstract (arXiv)

Agent self-improvement, where the backbone Large Language Model (LLM) of the agent are trained on trajectories sampled autonomously based on their own policies, has emerged as a promising approach for enhancing performance. Recent advancements, particularly in web environments, face a critical limitation: their performance will reach a stagnation point during autonomous learning cycles, hindering further improvement. We argue that this stems from limited exploration of the web environment and insufficient exploitation of pre-trained web knowledge in LLMs. To improve the performance of self-improvement, we propose a novel framework that introduces a co-evolving World Model LLM. This world model predicts the next observation based on the current observation and action within the web environment. Leveraging LLMs' pretrained knowledge of abundant web content, the World Model serves dual roles: (1) as a virtual web server generating self-instructed training data to continuously refine the agent's policy, and (2) as an imagination engine during inference, enabling look-ahead simulation to guide action selection for the agent LLM. Experiments in real-world web environments (Mind2Web-Live, WebVoyager, and GAIA-web) show a 10% performance gain over existing self-evolving agents, demonstrating the efficacy and generalizability of our approach, without using any distillation from more powerful close-sourced models. Our work establishes the necessity of integrating world models into autonomous agent frameworks to unlock sustained adaptability. Code is available at https://github.com/Tencent/SelfEvolvingAgent

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 42 of its 54 references that are in the map. First posted on arXiv 23 Apr 2025.

Builds on

42 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents10 Nov
  2. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation17 Oct
  3. 2024Agent Planning with World Knowledge Model23 May
  4. 2023Reasoning with Language Model is Planning with World Model24 May
  5. 2023Co-Learning Empirical Games and World Models23 May

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  11. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  12. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
25 more
  1. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  2. 2024Measuring short-form factuality in large language modelscited by 373
  3. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  4. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  5. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  6. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  7. 2024Tree Search for Language Model Agentscited by 151
  8. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  9. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  10. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 101
  11. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 81
  12. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
  13. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 62
  14. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 59
  15. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  16. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 55
  17. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 46
  18. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  19. 2025InSTA: Towards Internet-Scale Training For Agentscited by 32
  20. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvementcited by 29
  21. 2024B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasonerscited by 26
  22. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  23. 2025WebRollback: Enhancing Web Agents with Explicit Rollback Mechanismscited by 15
  24. 2024Cognitive Kernel: An Open-source Agent System towards Generalist Autopilotscited by 10
  25. 2025Streaming Looking Ahead with Token-level Self-rewardcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 45.

61 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2026SkillClaw: Let Skills Evolve Collectively with Agentic Evolvercited by 81
  3. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  5. 2025Scaling Agent Learning via Experience Synthesiscited by 44
  6. 2025Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Trainingcited by 42
  7. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  8. 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 26
  9. 2026K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Modelcited by 24
  10. 2026Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verificationcited by 22
  11. 2025WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesiscited by 21
  12. 2025DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seekingcited by 21
38 more
  1. 2025Test-Time Adaptation for LLM Agents via Environment Interactioncited by 20
  2. 2026Recursive Synthesis for Long-Horizon Terminal Taskscited by 19
  3. 2025LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Trainingcited by 18
  4. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 17
  5. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learningcited by 17
  6. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
  7. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  8. 2026Policy and World Modeling Co-Training for Language Agentscited by 15
  9. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  10. 2026DynaWeb: Model-Based Reinforcement Learning of Web Agentscited by 14
  11. 2026Generative Visual Code Mobile World Modelscited by 13
  12. 2026Self-Evolving World Models for LLM Agent Planningcited by 13
  13. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 12
  14. 2026SafePred: A Predictive Guardrail for Computer-Using Agents via World Modelscited by 12
  15. 2026World-Model-Augmented Web Agents with Action Correctioncited by 9
  16. 2026SEVerA: Verified Synthesis of Self-Evolving Agentscited by 8
  17. 2026Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesiscited by 8
  18. 2026OSExpert: Computer-Use Agents Learning Professional Skills via Explorationcited by 7
  19. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedbackcited by 6
  20. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6
  21. 2026CoMAP: Co-Evolving World Models and Agent Policies for LLM Agentscited by 6
  22. 2026Aligning Agentic World Models via Knowledgeable Experience Learningcited by 5
  23. 2026Table-as-Search: Formulate Long-Horizon Agentic Information Seeking as Table Completioncited by 5
  24. 2026Beyond State Consistency: Behavior Consistency in Text-Based World Modelscited by 5
  25. 2026AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learningcited by 5
  26. 2026SkillMAS: Skill Co-Evolution with LLM-based Multi-Agent Systemcited by 5
  27. 2026On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecificationcited by 4
  28. 2025Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agentscited by 3
  29. 2026Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Explorationcited by 3
  30. 2026Internalizing the Future: A Unified Agentic Training Paradigm for World Model Planningcited by 3
  31. 2026What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agentscited by 3
  32. 2026Self-evolving Embodied AIcited by 2
  33. 2026Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agentscited by 2
  34. 2026TAPO: Transition-Aware Policy Optimization for LLM Agentscited by 2
  35. 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agentscited by 2
  36. 2026Environment Maps: Structured Environmental Representations for Long-Horizon Agentscited by 1
  37. 2026SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoningcited by 1
  38. 2026See Better, Foresee Better, Act Wiser: Physically Grounded Proactive Modeling and Decision Makingcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents
  2. 2025AgentEvolver: Towards Efficient Self-Evolving Agent System
  3. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
  4. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills
  5. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agents
  6. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence
  7. 2026Safe and Scalable Web Agent Learning via Recreated Websites
  8. 2026WebXSkill: Skill Learning for Autonomous Web Agents