mlsanityData to 7 Oct 2026

An Illusion of Progress? Assessing the Current State of Web Agents

arXiv 2504.01382 ↗First posted 2 Apr 2025cs.AI cs.CLcited by 165

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

As digitalization and cloud technologies evolve, the web is becoming increasingly important in the modern society. Autonomous web agents based on large language models (LLMs) hold a great potential in work automation. It is therefore important to accurately measure and monitor the progression of their capabilities. In this work, we conduct a comprehensive and rigorous assessment of the current state of web agents. Our results depict a very different picture of the competency of current agents, suggesting over-optimism in previously reported results. This gap can be attributed to shortcomings in existing benchmarks. We introduce Online-Mind2Web, an online evaluation benchmark consisting of 300 diverse and realistic tasks spanning 136 websites. It enables us to evaluate web agents under a setting that approximates how real users use these agents. To facilitate more scalable evaluation and development, we also develop a novel LLM-as-a-Judge automatic evaluation method and show that it can achieve around 85% agreement with human judgment, substantially higher than existing methods. Finally, we present the first comprehensive comparative analysis of current web agents, highlighting both their strengths and limitations to inspire future research.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 56 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

42 of 56 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents17 Dec
  2. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials12 Dec
  3. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents10 Nov
  4. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning4 Nov
  5. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization25 Oct
  6. 2024Multimodal Auto Validation For Self-Refinement in Web Agents1 Oct
  7. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?22 Jul
  8. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems17 Jul
  9. 2024WebCanvas: Benchmarking Web Agents in Online Environments18 Jun
  10. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agent4 Apr
  11. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan
3 more
  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  2. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  8. 2023Cognitive Architectures for Language Agentscited by 422
  9. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  10. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  12. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
15 more
  1. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  2. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  3. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  4. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  5. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  6. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  7. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  8. 2024Tree Search for Language Model Agentscited by 148
  9. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  10. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  11. 2022A data-driven approach for learning to control computerscited by 87
  12. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  13. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
  14. 2023RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thoughtcited by 47
  15. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 44; 2026: 121.

165 papers.

Sort:

By area:

  1. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  2. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  3. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  4. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  5. 2025Agent Learning via Early Experiencecited by 79
  6. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 75
  7. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  8. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  9. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  10. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  11. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  12. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agentscited by 41
38 more
  1. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  2. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  3. 2025Measuring Agents in Productioncited by 35
  4. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  5. 2025Go-Browse: Training Web Agents with Structured Explorationcited by 32
  6. 2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Toolscited by 30
  7. 2025WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Taskscited by 29
  8. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  9. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  10. 2026ClawBench: Can AI Agents Complete Everyday Online Tasks?cited by 28
  11. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 26
  12. 2025PolySkill: Learning Generalizable Skills Through Polymorphic Abstractioncited by 25
  13. 2026WebXSkill: Skill Learning for Autonomous Web Agentscited by 24
  14. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 20
  15. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  16. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 19
  17. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 19
  18. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 18
  19. 2025Build the web for agents, not agents for the webcited by 17
  20. 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 17
  21. 2025WebMall -- A Multi-Shop Benchmark for Evaluating Web Agentscited by 15
  22. 2025WALT: Web Agents that Learn Toolscited by 15
  23. 2025Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agentscited by 14
  24. 2025Scaling Agents for Computer Usecited by 14
  25. 2025DECEPTICON: How Dark Patterns Manipulate Web Agentscited by 13
  26. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  27. 2025SynthAgent: Adapting Web Agents with Synthetic Supervisioncited by 12
  28. 2025Prune4Web: DOM Tree Pruning Programming for Web Agentcited by 11
  29. 2025WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filteringcited by 11
  30. 2026The Art of Building Verifiers for Computer Use Agentscited by 11
  31. 2026Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflowscited by 11
  32. 2025WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Qualitycited by 10
  33. 2026Safe and Scalable Web Agent Learning via Recreated Websitescited by 9
  34. 2026Efficient Benchmarking of AI Agentscited by 9
  35. 2025Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directednesscited by 8
  36. 2025SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agentscited by 8
  37. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deploymentcited by 8
  38. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8

50 most cited of 165 All 165 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
  2. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
  3. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
  4. 2024WebSuite: Systematically Evaluating Why Web Agents Fail
  5. 2024The BrowserGym Ecosystem for Web Agent Research
  6. 2024Towards a Realistic Long-Term Benchmark for Open-Web Research Agents
  7. 2025Deep Research Bench: Evaluating AI Web Research Agents
  8. 2025BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks