# ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues? arXiv: https://arxiv.org/abs/2507.05639 · HTML: https://mlsanity.com/paper/2507.05639/ ## Key facts - Cited by 27 papers in the map (data through 14 Oct 2026). - Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. - Builds on 9 of its 30 references that are in the map. - First posted on arXiv 8 Jul 2025. ## Placed in - LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/) ## Builds on (most cited first, 9 of 9) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022) - [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020) - [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018) - [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024) - [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024) - [Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizations](https://mlsanity.com/paper/2310.05421/) (2023) - [Speech Aware Dialog System Technology Challenge (DSTC11)](https://mlsanity.com/paper/2212.08704/) (2022) - [EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association](https://mlsanity.com/paper/2505.15196/) (2025) ## Built on by (most cited first, 27 of 27) - [Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations](https://mlsanity.com/paper/2601.17087/) (2026) - [WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents](https://mlsanity.com/paper/2508.13024/) (2025) - [EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce](https://mlsanity.com/paper/2512.08868/) (2025) - [MemOrb: A Plug-and-Play Verbal-Reinforcement Memory Layer for E-Commerce Customer Service](https://mlsanity.com/paper/2509.18713/) (2025) - [SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue](https://mlsanity.com/paper/2602.03548/) (2026) - [CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems](https://mlsanity.com/paper/2604.12312/) (2026) - [MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations](https://mlsanity.com/paper/2607.28956/) (2026) - [Optimizing Agentic Workflows using Meta-tools](https://mlsanity.com/paper/2601.22037/) (2026) - [Survey of Specialized Large Language Model](https://mlsanity.com/paper/2508.19667/) (2025) - [Talk Less, Call Right: Enhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Prompting](https://mlsanity.com/paper/2509.00482/) (2025) - [EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce](https://mlsanity.com/paper/2601.02752/) (2026) - [Mix-Ecom: Towards Mixed-Type E-Commerce Dialogues with Complex Domain Rules](https://mlsanity.com/paper/2509.23836/) (2025) - [GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents](https://mlsanity.com/paper/2609.12191/) (2026) - [JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks](https://mlsanity.com/paper/2602.06486/) (2026) - [What Makes a Sale? Simulating End-to-End Seller--Buyer Retail Dynamics with LLM Agents](https://mlsanity.com/paper/2604.04468/) (2026) - [Benchmarking and Learning Real-World Customer Service Dialogue](https://mlsanity.com/paper/2510.22143/) (2025) - [AEGIS: Exploring the Limit of World Knowledge Capabilities for Unified Mulitmodal Models](https://mlsanity.com/paper/2601.00561/) (2026) - [Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue](https://mlsanity.com/paper/2602.22697/) (2026) - [GAIN: A Benchmark for Goal-Aligned Decision-Making of Large Language Models under Imperfect Norms](https://mlsanity.com/paper/2603.18469/) (2026) - [Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvement](https://mlsanity.com/paper/2604.09579/) (2026) - [Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports](https://mlsanity.com/paper/2605.17561/) (2026) - [From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service](https://mlsanity.com/paper/2603.23172/) (2026) - [From Triggers to Emotions: A CPM-Grounded Appraisal Multi-Agent for Dynamic Emotional Evolution in Persona-Based Dialogue](https://mlsanity.com/paper/2607.07824/) (2026) - [SkillEvo: Self-Renewing Evolution Gradients from Multi-Turn Interaction Feedback](https://mlsanity.com/paper/2608.13120/) (2026) - [PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants](https://mlsanity.com/paper/2608.26180/) (2026) - [From Monolithic Blending to Agentic Orchestration: Dynamic Response for Conversational Assistants at Scale](https://mlsanity.com/paper/2609.05758/) (2026) - [PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents](https://mlsanity.com/paper/2609.31468/) (2026) ## Abstract (arXiv) In this paper, we introduce ECom-Bench, the first benchmark framework for evaluating LLM agent with multimodal capabilities in the e-commerce customer support domain. ECom-Bench features dynamic user simulation based on persona information collected from real e-commerce customer interactions and a realistic task dataset derived from authentic e-commerce dialogues. These tasks, covering a wide range of business scenarios, are designed to reflect real-world complexities, making ECom-Bench highly challenging. For instance, even advanced models like GPT-4o achieve only a 10-20% pass^3 metric in our benchmark, highlighting the substantial difficulties posed by complex e-commerce scenarios. The code and data have been made publicly available at https://github.com/XiaoduoAILab/ECom-Bench to facilitate further research and development in this domain.