mlsanityData to 7 Oct 2026

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

arXiv 2604.08516 ↗First posted 9 Apr 2026cs.CVcited by 17

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Web agents--autonomous systems that navigate and execute tasks on the web on behalf of users--have the potential to transform how people interact with the digital world. However, the most capable web agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open multimodal web agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb agents operate as instruction-conditioned visual-language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM) agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web agents.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 43 of its 64 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

43 of 64 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machines15 Feb
  2. 2025Fara-7B: An Efficient Agentic Model for Computer Use24 Nov
  3. 2025An Illusion of Progress? Assessing the Current State of Web Agents2 Apr
  4. 2024The BrowserGym Ecosystem for Web Agent Research6 Dec
  5. 2024Beyond Browsing: API-Based Web Agents21 Oct
  6. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?22 Jul
  7. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems17 Jul
  8. 2024WebCanvas: Benchmarking Web Agents in Online Environments18 Jun
  9. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?12 Mar
  10. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan
  12. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
2 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
  2. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 29

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  6. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  8. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  9. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  10. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  11. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  12. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
17 more
  1. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  2. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  4. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 221
  5. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  6. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  7. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  8. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  9. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  10. 2024Tree Search for Language Model Agentscited by 148
  11. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  12. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  13. 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 81
  14. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  15. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  16. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  17. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception
  2. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
  3. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
  4. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
  5. 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
  6. 2024MMInA: Benchmarking Multihop Multimodal Internet Agents
  7. 2025Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights
  8. 2025BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions