mlsanityData to 7 Oct 2026

AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?

arXiv 2407.15711 ↗First posted 22 Jul 2024cs.CLcited by 155

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses. We introduce AssistantBench, a challenging new benchmark consisting of 214 realistic tasks that can be automatically evaluated, covering different scenarios and domains. We find that AssistantBench exposes the limitations of current systems, including language models and retrieval-augmented language models, as no model reaches an accuracy of more than 26 points. While closed-book LMs perform well in terms of accuracy, they exhibit low precision and tend to hallucinate facts. State-of-the-art web agents reach a score of near zero. Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly outperforms previous agents, and an ensemble of SPA and closed-book models reaches the best overall performance. Moreover, we analyze failures of current systems and highlight that open web navigation remains a major challenge.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 52 of its 67 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

52 of 67 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024MMInA: Benchmarking Multihop Multimodal Internet Agents15 Apr
  2. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agents18 Mar
  3. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?12 Mar
  4. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan
  6. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 45

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  12. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
33 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  10. 2019Green AIcited by 620
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  13. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  14. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  15. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  16. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  17. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  18. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  19. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  20. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  21. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  22. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  23. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  24. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  25. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  26. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  27. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  28. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  29. 2022A data-driven approach for learning to control computerscited by 87
  30. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  31. 2018Learning to Navigate the Webcited by 60
  32. 2021How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AIcited by 22
  33. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 58; 2026: 87.

155 papers.

Sort:

By area:

  1. 2024Agent Workflow Memorycited by 342
  2. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  3. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  4. 2025Survey on Evaluation of LLM-based Agentscited by 190
  5. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  6. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  7. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  8. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  9. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  10. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  11. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  12. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
38 more
  1. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  2. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  3. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  4. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
  5. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  6. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  7. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  8. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 75
  9. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  10. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  11. 2024MindSearch: Mimicking Human Minds Elicits Deep AI Searchercited by 65
  12. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 58
  13. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 43
  14. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  15. 2025BEARCUBS: A benchmark for computer-using web agentscited by 29
  16. 2025WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Taskscited by 29
  17. 2026ClawBench: Can AI Agents Complete Everyday Online Tasks?cited by 28
  18. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  19. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  20. 2025Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attributioncited by 25
  21. 2025FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecastingcited by 23
  22. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 23
  23. 2026Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluationcited by 21
  24. 2025MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documentscited by 20
  25. 2024Infogent: An Agent-Based Framework for Web Information Aggregationcited by 19
  26. 2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systemscited by 19
  27. 2025BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actionscited by 17
  28. 2026From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systemscited by 17
  29. 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 17
  30. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 17
  31. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  32. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  33. 2025The AI Agent Indexcited by 14
  34. 2025Abduct, Act, Predict: Scaffolding Causal Inference for Automated Failure Attribution in Multi-Agent Systemscited by 14
  35. 2025FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysiscited by 13
  36. 2024From Grounding to Planning: Benchmarking Bottlenecks in Web Agentscited by 12
  37. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROIcited by 12
  38. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12

50 most cited of 155 All 155 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System
  2. 2025Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights
  3. 2025MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web (Technical Report)
  4. 2026DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents
  5. 2023OpenAgents: An Open Platform for Language Agents in the Wild
  6. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
  7. 2025Build the web for agents, not agents for the web
  8. 2026DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Tasks