# WAREX: Web Agent Reliability Evaluation on Existing Benchmarks

arXiv: https://arxiv.org/abs/2510.03285 · HTML: https://mlsanity.com/paper/2510.03285/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 22 of 22)

- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://mlsanity.com/paper/2307.14539/) (2023)
- [Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents](https://mlsanity.com/paper/2408.07199/) (2024)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [Image Hijacks: Adversarial Images can Control Generative Models at Runtime](https://mlsanity.com/paper/2309.00236/) (2023)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents](https://mlsanity.com/paper/2410.06703/) (2024)
- [AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories](https://mlsanity.com/paper/2504.08942/) (2025)
- [SafeArena: Evaluating the Safety of Autonomous Web Agents](https://mlsanity.com/paper/2503.04957/) (2025)
- [WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks](https://mlsanity.com/paper/2407.05291/) (2024)
- [SteP: Stacked LLM Policies for Web Actions](https://mlsanity.com/paper/2310.03720/) (2023)
- [REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites](https://mlsanity.com/paper/2504.11543/) (2025)
- [DoomArena: A framework for Testing AI Agents Against Evolving Security Threats](https://mlsanity.com/paper/2504.14064/) (2025)
- [From Grounding to Planning: Benchmarking Bottlenecks in Web Agents](https://mlsanity.com/paper/2409.01927/) (2024)

## Built on by (most cited first, 15 of 15)

- [Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond](https://mlsanity.com/paper/2604.22748/) (2026)
- [Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures](https://mlsanity.com/paper/2607.28802/) (2026)
- [Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents](https://mlsanity.com/paper/2606.15017/) (2026)
- [ClawTrap: A MITM-Based Red-Teaming Framework for Real-World OpenClaw Security Evaluation](https://mlsanity.com/paper/2603.18762/) (2026)
- [Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments](https://mlsanity.com/paper/2608.02670/) (2026)
- [Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents](https://mlsanity.com/paper/2609.29095/) (2026)
- [StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability](https://mlsanity.com/paper/2604.16385/) (2026)
- [Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents](https://mlsanity.com/paper/2605.19149/) (2026)
- [Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluation](https://mlsanity.com/paper/2608.26195/) (2026)
- [Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions](https://mlsanity.com/paper/2607.21635/) (2026)
- [CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception](https://mlsanity.com/paper/2608.08392/) (2026)
- [Web Agents Should Use Typed Actions Instead of Click-Based Browsing](https://mlsanity.com/paper/2602.17245/) (2026)
- [Signal-Driven Observation for Long-Horizon Web Agents](https://mlsanity.com/paper/2606.06708/) (2026)
- [Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents](https://mlsanity.com/paper/2610.01348/) (2026)
- [AgentStripes: An Evidence-Graded Maturity Model for AI Agents](https://mlsanity.com/paper/2610.18603/) (2026)

## Abstract (arXiv)

Recent advances in browser-based LLM agents have shown promise for automating tasks ranging from simple form filling to hotel booking or online shopping. Current benchmarks measure agent performance in controlled environments, such as containers or stable networks, where websites behave deterministically. However, in the real world, users access websites over networks and HTTPS connections that introduce instability from multiple sources: client-side, server-side issues or broader system failures. Moreover, live websites are prone to web attacks such Cross-Site Scripting, as well as general site modifications which can cause unexpected or malicious pop-ups or improper functionality. To address this gap, we present WAREX: Web Agent Reliability Evaluation on Existing Benchmarks. We measure the impact of WAREX across three popular benchmarks: WebArena, WebVoyager, and REAL. Our experiments show that introducing WAREX leads to significant drops in task success rates, highlighting the limited robustness of state-of-the-art agents.