# BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks arXiv: https://arxiv.org/abs/2510.02418 ยท HTML: https://mlsanity.com/paper/2510.02418/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. - Builds on 23 of its 37 references that are in the map. - First posted on arXiv 2 Oct 2025. ## Placed in - LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/) ## Builds on (most cited first, 23 of 23) - [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022) - [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024) - [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024) - [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023) - [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024) - [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025) - [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024) - [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024) - [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024) - [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025) - [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024) - [AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?](https://mlsanity.com/paper/2407.15711/) (2024) - [WebQA: Multihop and Multimodal QA](https://mlsanity.com/paper/2109.00590/) (2021) - [MMInA: Benchmarking Multihop Multimodal Internet Agents](https://mlsanity.com/paper/2404.09992/) (2024) - [VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks](https://mlsanity.com/paper/2410.19100/) (2024) - [Search Arena: Analyzing Search-Augmented LLMs](https://mlsanity.com/paper/2506.05334/) (2025) - [WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks](https://mlsanity.com/paper/2506.01952/) (2025) - [Copilot Arena: A Platform for Code LLM Evaluation in the Wild](https://mlsanity.com/paper/2502.09328/) (2025) - [Adaptively profiling models with task elicitation](https://mlsanity.com/paper/2503.01986/) (2025) - [Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction](https://mlsanity.com/paper/2502.17541/) (2025) ## Built on by (most cited first, 14 of 14) - [PPTArena: A Benchmark for PowerPoint Editing](https://mlsanity.com/paper/2512.03042/) (2025) - [Log analysis is necessary for credible evaluation of AI agents](https://mlsanity.com/paper/2605.08545/) (2026) - [Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective](https://mlsanity.com/paper/2603.14248/) (2026) - [AgentBay: A Hybrid Interaction Sandbox for Seamless Human-AI Intervention in Agentic Systems](https://mlsanity.com/paper/2512.04367/) (2025) - [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026) - [WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark](https://mlsanity.com/paper/2604.10988/) (2026) - [Open-World Evaluations for Measuring Frontier AI Capabilities](https://mlsanity.com/paper/2605.20520/) (2026) - [Gym-V: A Unified Vision Environment System for Agentic Vision Research](https://mlsanity.com/paper/2603.15432/) (2026) - [StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability](https://mlsanity.com/paper/2604.16385/) (2026) - [From Task Solving to Robust Real-World Adaptation in LLM Agents](https://mlsanity.com/paper/2602.02760/) (2026) - [STRUCTUREDAGENT: Planning with AND/OR Trees for Long-Horizon Web Tasks](https://mlsanity.com/paper/2603.05294/) (2026) - [Failing to Explore: Language Models on Interactive Tasks](https://mlsanity.com/paper/2601.22345/) (2026) - [WebRider: Persona-Conditioned Intent Controllers for Live-Web Assistance](https://mlsanity.com/paper/2608.06704/) (2026) - [AgentStripes: An Evidence-Graded Maturity Model for AI Agents](https://mlsanity.com/paper/2610.18603/) (2026) ## Abstract (arXiv) LLM web agents now browse and take actions on the open web, yet current agent evaluations are constrained to sandboxed environments or artificial tasks. We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks, runs Arena-style head-to-head comparisons, and uses step-level human feedback to surface failure modes. Collecting and analyzing step-level annotations on the agent traces, we identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs. By constructing targeted datasets to further study these tasks, we discover variations in how different language models navigate these failure modes. We find, for example, that o4-mini deploys a wider variety of strategies to circumvent captcha resolution than other models and DeepSeek-R1 consistently misleads users about pop-up banner closure. Our findings surface both the diversity and brittleness of current web agents. More broadly, our benchmarking methodology provides an approach to evaluating and understanding web agent failure modes at scale.