mlsanityData to 14 Oct 2026

WebCanvas: Benchmarking Web Agents in Online Environments

arXiv 2406.12373 ↗First posted 18 Jun 2024cs.CL cs.AI cs.LGcited by 121

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

For web agents to be practically useful, they must adapt to the continuously evolving web environment characterized by frequent updates to user interfaces and content. However, most existing benchmarks only capture the static aspects of the web. To bridge this gap, we introduce WebCanvas, an innovative online evaluation framework for web agents that effectively addresses the dynamic nature of web interactions. WebCanvas contains three main components to facilitate realistic assessments: (1) A novel evaluation metric which reliably capture critical intermediate actions or states necessary for task completions while disregarding noise caused by insignificant events or changed web-elements. (2) A benchmark dataset called Mind2Web-Live, a refined version of original Mind2Web static dataset containing 542 tasks with 2439 intermediate evaluation states; (3) Lightweight and generalizable annotation tools and testing pipelines that enables the community to collect and maintain the high-quality, up-to-date dataset. Building on WebCanvas, we open-source an agent framework with extensible modules for reasoning, providing a foundation for the community to conduct online inference and evaluations. Our best-performing agent achieves a task success rate of 23.1% and a task completion rate of 48.8% on the Mind2Web-Live test set. Additionally, we analyze the performance discrepancies across various websites, domains, and experimental environments. We encourage the community to contribute further insights on online agent evaluation, thereby advancing this field of research.

Key facts. Cited by 121 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 48 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

36 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agent4 Apr
  2. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?12 Mar
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  4. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  6. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
17 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  6. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  7. 2023Cognitive Architectures for Language Agentscited by 429
  8. 2023Language Models can Solve Computer Taskscited by 372
  9. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  10. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  11. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  12. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  13. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  14. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  15. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  16. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  17. 2023Hierarchical Prompting Assists Large Language Model on Web Navigationcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 51; 2026: 54.

121 papers.

Sort:

By area:

  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  2. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  3. 2024AI Agents That Mattercited by 219
  4. 2025Survey on Evaluation of LLM-based Agentscited by 204
  5. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
  6. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  7. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  8. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  9. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  10. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  11. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  12. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 120
38 more
  1. 2025Inducing Programmatic Skills for Agentic Taskscited by 111
  2. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 106
  3. 2024GUI Agents: A Surveycited by 103
  4. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 102
  5. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 76
  6. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
  7. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  8. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 61
  9. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  10. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  11. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  12. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 49
  13. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 46
  14. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 46
  15. 2025Fully Autonomous AI Agents Should Not be Developedcited by 45
  16. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  17. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  18. 2025CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigationcited by 36
  19. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  20. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 33
  21. 2026ClawBench: Can AI Agents Complete Everyday Online Tasks?cited by 33
  22. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 32
  23. 2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Toolscited by 32
  24. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 31
  25. 2025WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Taskscited by 30
  26. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  27. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 24
  28. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  29. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 21
  30. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  31. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 19
  32. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  33. 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 19
  34. 2025Build the web for agents, not agents for the webcited by 18
  35. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 17
  36. 2024Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understandingcited by 16
  37. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  38. 2025Evaluating Long-Context Reasoning in LLM-Based WebAgentscited by 15

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
  2. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
  3. 2024From Grounding to Planning: Benchmarking Bottlenecks in Web Agents
  4. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
  5. 2025BEARCUBS: A benchmark for computer-using web agents
  6. 2024WebSuite: Systematically Evaluating Why Web Agents Fail
  7. 2026StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability
  8. 2026Revisiting Observation Reduction for Web Agents: Comprehensive Evaluation with a Lightweight Framework