# OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization

arXiv: https://arxiv.org/abs/2410.19609 · HTML: https://mlsanity.com/paper/2410.19609/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 22 of 22)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents](https://mlsanity.com/paper/2408.07199/) (2024)
- [Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents](https://mlsanity.com/paper/2403.02502/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [AgentGym: Evolving Large Language Model-based Agents across Diverse Environments](https://mlsanity.com/paper/2406.04151/) (2024)
- [WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration](https://mlsanity.com/paper/2408.15978/) (2024)
- [Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems](https://mlsanity.com/paper/2407.13032/) (2024)
- [LASER: LLM Agent with State-Space Exploration for Web Navigation](https://mlsanity.com/paper/2309.08172/) (2023)
- [ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning](https://mlsanity.com/paper/2410.02052/) (2024)
- [Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models](https://mlsanity.com/paper/2402.12058/) (2024)
- [Large Language Models Can Self-Improve At Web Agent Tasks](https://mlsanity.com/paper/2405.20309/) (2024)
- [Bootstrapping Vision-Language Learning with Decoupled Language Pre-training](https://mlsanity.com/paper/2307.07063/) (2023)
- [Dual-View Visual Contextualization for Web Navigation](https://mlsanity.com/paper/2402.04476/) (2024)
- [WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents](https://mlsanity.com/paper/2404.05902/) (2024)
- [Cognitive Kernel: An Open-source Agent System towards Generalist Autopilots](https://mlsanity.com/paper/2409.10277/) (2024)

## Built on by (most cited first, 30 of 46)

- [Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2503.09572/) (2025)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025)
- [WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2505.16421/) (2025)
- [OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis](https://mlsanity.com/paper/2412.19723/) (2024)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents](https://mlsanity.com/paper/2502.11357/) (2025)
- [WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model](https://mlsanity.com/paper/2504.21024/) (2025)
- [Self-Challenging Language Model Agents](https://mlsanity.com/paper/2506.01716/) (2025)
- [Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents](https://mlsanity.com/paper/2412.13194/) (2024)
- [DeepShop: A Benchmark for Deep Research Shopping Agents](https://mlsanity.com/paper/2506.02839/) (2025)
- [Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training](https://mlsanity.com/paper/2508.00414/) (2025)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use](https://mlsanity.com/paper/2508.04482/) (2025)
- [WebXSkill: Skill Learning for Autonomous Web Agents](https://mlsanity.com/paper/2604.13318/) (2026)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback](https://mlsanity.com/paper/2505.20013/) (2025)
- [SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning](https://mlsanity.com/paper/2605.09423/) (2026)
- [WebRollback: Enhancing Web Agents with Explicit Rollback Mechanisms](https://mlsanity.com/paper/2504.11788/) (2025)
- [SynthAgent: Adapting Web Agents with Synthetic Supervision](https://mlsanity.com/paper/2511.06101/) (2025)
- [Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback](https://mlsanity.com/paper/2506.04287/) (2025)
- [WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering](https://mlsanity.com/paper/2512.10962/) (2025)
- [OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis](https://mlsanity.com/paper/2604.15093/) (2026)
- [Watch and Learn: Learning to Use Computers from Online Videos](https://mlsanity.com/paper/2510.04673/) (2025)
- [SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks](https://mlsanity.com/paper/2604.08865/) (2026)
- [WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning](https://mlsanity.com/paper/2505.22942/) (2025)
- [OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents](https://mlsanity.com/paper/2606.02031/) (2026)
- [Fara-1.5: Scalable Learning Environments for Computer Use Agents](https://mlsanity.com/paper/2606.20785/) (2026)
- [The Tool Illusion: Rethinking Tool Use in Web Agents](https://mlsanity.com/paper/2604.03465/) (2026)
- [WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents](https://mlsanity.com/paper/2504.12682/) (2025)

## Abstract (arXiv)

The rapid development of large language and multimodal models has sparked significant interest in using proprietary models, such as GPT-4o, to develop autonomous agents capable of handling real-world scenarios like web navigation. Although recent open-source efforts have tried to equip agents with the ability to explore environments and continuously improve over time, they are building text-only agents in synthetic environments where the reward signals are clearly defined. Such agents struggle to generalize to realistic settings that require multimodal perception abilities and lack ground-truth signals. In this paper, we introduce an open-source framework designed to facilitate the development of multimodal web agent that can autonomously conduct real-world exploration and improve itself. We first train the base model with imitation learning to gain the basic abilities. We then let the agent explore the open web and collect feedback on its trajectories. After that, it further improves its policy by learning from well-performing trajectories judged by another general-purpose model. This exploration-feedback-optimization cycle can continue for several iterations. Experimental results show that our web agent successfully improves itself after each iteration, demonstrating strong performance across multiple test sets.