# EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce

arXiv: https://arxiv.org/abs/2601.02752 · HTML: https://mlsanity.com/paper/2601.02752/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://mlsanity.com/paper/2306.02224/) (2023)
- [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://mlsanity.com/paper/2411.10442/) (2024)
- [A Survey on Large Language Model-Based Game Agents](https://mlsanity.com/paper/2404.02039/) (2024)
- [A Survey on Code Generation with LLM-based Agents](https://mlsanity.com/paper/2508.00083/) (2025)
- [MiniCPM4: Ultra-Efficient LLMs on End Devices](https://mlsanity.com/paper/2506.07900/) (2025)
- [ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?](https://mlsanity.com/paper/2507.05639/) (2025)
- [U-NEED: A Fine-grained Dataset for User Needs-Centric E-commerce Conversational Recommendation](https://mlsanity.com/paper/2305.04774/) (2023)
- [EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association](https://mlsanity.com/paper/2505.15196/) (2025)
- [Cite Before You Speak: Enhancing Context-Response Grounding in E-commerce Conversational LLM-Agents](https://mlsanity.com/paper/2503.04830/) (2025)
- [MemOrb: A Plug-and-Play Verbal-Reinforcement Memory Layer for E-Commerce Customer Service](https://mlsanity.com/paper/2509.18713/) (2025)
- [Mix-Ecom: Towards Mixed-Type E-Commerce Dialogues with Complex Domain Rules](https://mlsanity.com/paper/2509.23836/) (2025)
- [EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization](https://mlsanity.com/paper/2408.02970/) (2024)
- [Experimental Evaluation of Machine Learning Models for Goal-oriented Customer Service Chatbot with Pipeline Architecture](https://mlsanity.com/paper/2409.18568/) (2024)

## Built on by (most cited first, 4 of 4)

- [MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations](https://mlsanity.com/paper/2607.28956/) (2026)
- [What Makes a Sale? Simulating End-to-End Seller--Buyer Retail Dynamics with LLM Agents](https://mlsanity.com/paper/2604.04468/) (2026)
- [FlexComp: One Model for Every Ratio in Context Compression](https://mlsanity.com/paper/2609.11192/) (2026)
- [RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce](https://mlsanity.com/paper/2609.38974/) (2026)

## Abstract (arXiv)

Large Language Model (LLM)-based agents are increasingly deployed in e-commerce applications to assist customer services in tasks such as product inquiries, recommendations, and order management. Existing benchmarks primarily evaluate whether these agents successfully complete the final task, overlooking the intermediate reasoning stages that are crucial for effective decision-making. To address this gap, we propose EComStage, a unified benchmark for evaluating agent-capable LLMs across the comprehensive stage-wise reasoning process: Perception (understanding user intent), Planning (formulating an action plan), and Action (executing the decision). EComStage evaluates LLMs through seven separate representative tasks spanning diverse e-commerce scenarios, with all samples human-annotated and quality-checked. Unlike prior benchmarks that focus only on customer-oriented interactions, EComStage also evaluates merchant-oriented scenarios, including promotion management, content review, and operational support relevant to real-world applications. We evaluate a wide range of over 30 LLMs, spanning from 1B to over 200B parameters, including open-source models and closed-source APIs, revealing stage/orientation-specific strengths and weaknesses. Our results provide fine-grained, actionable insights for designing and optimizing LLM-based agents in real-world e-commerce settings.