# WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents arXiv: https://arxiv.org/abs/2508.13024 ยท HTML: https://mlsanity.com/paper/2508.13024/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. - Builds on 17 of its 30 references that are in the map. - First posted on arXiv 18 Aug 2025. ## Placed in - LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/) ## Builds on (most cited first, 17 of 17) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022) - [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023) - [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025) - [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025) - [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025) - [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024) - [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025) - [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025) - [DeepShop: A Benchmark for Deep Research Shopping Agents](https://mlsanity.com/paper/2506.02839/) (2025) - [REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites](https://mlsanity.com/paper/2504.11543/) (2025) - [BEARCUBS: A benchmark for computer-using web agents](https://mlsanity.com/paper/2503.07919/) (2025) - [WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks](https://mlsanity.com/paper/2506.01952/) (2025) - [ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents](https://mlsanity.com/paper/2508.04266/) (2025) - [ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?](https://mlsanity.com/paper/2507.05639/) (2025) ## Built on by (most cited first, 17 of 17) - [AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping](https://mlsanity.com/paper/2602.12315/) (2026) - [MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web (Technical Report)](https://mlsanity.com/paper/2511.23281/) (2025) - [HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application](https://mlsanity.com/paper/2510.19631/) (2025) - [ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents](https://mlsanity.com/paper/2605.16116/) (2026) - [SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce](https://mlsanity.com/paper/2602.01443/) (2026) - [Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agents](https://mlsanity.com/paper/2509.12049/) (2025) - [StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability](https://mlsanity.com/paper/2604.16385/) (2026) - [ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons](https://mlsanity.com/paper/2608.09282/) (2026) - [SAGE: Steerable Agentic Data Generation for Deep Search with Execution Feedback](https://mlsanity.com/paper/2601.18202/) (2026) - [SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents](https://mlsanity.com/paper/2605.19219/) (2026) - [Agentic Commerce World: An Auditable and Verifiable Environment for Vibe Commerce](https://mlsanity.com/paper/2608.02441/) (2026) - [RiskWebWorld: A Realistic Interactive Benchmark for GUI Agents in E-commerce Risk Management](https://mlsanity.com/paper/2604.13531/) (2026) - [RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents](https://mlsanity.com/paper/2605.18805/) (2026) - [Beyond expert users: agents should help users construct preferences, not just elicit them](https://mlsanity.com/paper/2606.30863/) (2026) - [Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents](https://mlsanity.com/paper/2607.22689/) (2026) - [Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification](https://mlsanity.com/paper/2609.16093/) (2026) - [Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce](https://mlsanity.com/paper/2610.17378/) (2026) ## Abstract (arXiv) LLM-based web agents have the potential to automate long-running web tasks, such as searching for products in multiple e-shops and subsequently ordering the cheapest products that meet the users needs. Benchmarks for evaluating web agents either require agents to perform tasks online using the live Web or offline using simulated environments, the latter allowing for the exact reproduction of the experimental setup. While DeepShop and ShoppingComp provide online benchmarks that require agents to perform challenging shopping tasks, existing offline benchmarks such as WebShop, WebArena, and Mind2Web cover only comparatively simple e-commerce tasks performed against a single shop containing product data from a single source. What is missing is an e-commerce benchmark that simulates multiple shops containing heterogeneous product data and requires agents to perform complex retrieval tasks. We fill this gap by introducing WebMall, the first offline multi-shop benchmark for evaluating web agents on challenging comparison shopping tasks. WebMall consists of four simulated shops populated with product data extracted from the Common Crawl. The WebMall tasks range from specific product searches and price comparisons to advanced searches for complementary or substitute products, as well as checkout processes. We validate WebMall using eight agents that differ in observation space, availability of short-term memory, and the employed LLM. The validation highlights the difficulty of the benchmark, with the best-performing agents achieving task completion rates below 65% in the task categories cheapest product search and vague product search.