mlsanityData to 14 Oct 2026

ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?

arXiv 2507.05639 ↗First posted 8 Jul 2025cs.CLcited by 27

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

In this paper, we introduce ECom-Bench, the first benchmark framework for evaluating LLM agent with multimodal capabilities in the e-commerce customer support domain. ECom-Bench features dynamic user simulation based on persona information collected from real e-commerce customer interactions and a realistic task dataset derived from authentic e-commerce dialogues. These tasks, covering a wide range of business scenarios, are designed to reflect real-world complexities, making ECom-Bench highly challenging. For instance, even advanced models like GPT-4o achieve only a 10-20% pass^3 metric in our benchmark, highlighting the substantial difficulties posed by complex e-commerce scenarios. The code and data have been made publicly available at https://github.com/XiaoduoAILab/ECom-Bench to facilitate further research and development in this domain.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 9 of its 30 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

9 of 30 references.

References · 9

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  2. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  4. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  5. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  6. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  7. 2023Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizationscited by 61
  8. 2022Speech Aware Dialog System Technology Challenge (DSTC11)cited by 10
  9. 2025EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Associationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 20.

27 papers.

Sort:

By area:

  1. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 40
  2. 2025WebMall -- A Multi-Shop Benchmark for Evaluating Web Agentscited by 17
  3. 2025EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commercecited by 8
  4. 2025MemOrb: A Plug-and-Play Verbal-Reinforcement Memory Layer for E-Commerce Customer Servicecited by 7
  5. 2026SEAD: Self-Evolving Agent for Multi-Turn Service Dialoguecited by 7
  6. 2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 6
  7. 2026MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operationscited by 6
  8. 2026Optimizing Agentic Workflows using Meta-toolscited by 5
  9. 2025Survey of Specialized Large Language Modelcited by 4
  10. 2025Talk Less, Call Right: Enhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Promptingcited by 4
  11. 2026EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commercecited by 4
  12. 2025Mix-Ecom: Towards Mixed-Type E-Commerce Dialogues with Complex Domain Rulescited by 3
15 more
  1. 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agentscited by 3
  2. 2026JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Taskscited by 2
  3. 2026What Makes a Sale? Simulating End-to-End Seller--Buyer Retail Dynamics with LLM Agentscited by 2
  4. 2025Benchmarking and Learning Real-World Customer Service Dialoguecited by 1
  5. 2026AEGIS: Exploring the Limit of World Knowledge Capabilities for Unified Mulitmodal Modelscited by 1
  6. 2026Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialoguecited by 1
  7. 2026GAIN: A Benchmark for Goal-Aligned Decision-Making of Large Language Models under Imperfect Normscited by 1
  8. 2026Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvementcited by 1
  9. 2026Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reportscited by 1
  10. 2026From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Servicecited by 0
  11. 2026From Triggers to Emotions: A CPM-Grounded Appraisal Multi-Agent for Dynamic Emotional Evolution in Persona-Based Dialoguecited by 0
  12. 2026SkillEvo: Self-Renewing Evolution Gradients from Multi-Turn Interaction Feedbackcited by 0
  13. 2026PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistantscited by 0
  14. 2026From Monolithic Blending to Agentic Orchestration: Dynamic Response for Conversational Assistants at Scalecited by 0
  15. 2026PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation
  2. 2025MindFlow: Revolutionizing E-commerce Customer Support with Multimodal LLM Agents
  3. 2026Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence
  4. 2026Shopping Companion: Benchmarking and Training LLM Agents for Long-Horizon Preference-Grounded E-Commerce Tasks
  5. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications
  6. 2026RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
  7. 2026CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
  8. 2025ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents