# MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

arXiv: https://arxiv.org/abs/2607.28956 · HTML: https://mlsanity.com/paper/2607.28956/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 22 of 22)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows](https://mlsanity.com/paper/2508.09124/) (2025)
- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents](https://mlsanity.com/paper/2508.04266/) (2025)
- [ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?](https://mlsanity.com/paper/2507.05639/) (2025)
- [UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios](https://mlsanity.com/paper/2509.21766/) (2025)
- [Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks](https://mlsanity.com/paper/2604.24964/) (2026)
- [ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants](https://mlsanity.com/paper/2601.18225/) (2026)
- [AI Agents for Inventory Control: Human-LLM-OR Complementarity](https://mlsanity.com/paper/2602.12631/) (2026)
- [CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies](https://mlsanity.com/paper/2606.16613/) (2026)
- [A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains](https://mlsanity.com/paper/2508.15832/) (2025)
- [HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds](https://mlsanity.com/paper/2508.12782/) (2025)
- [RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments](https://mlsanity.com/paper/2606.15862/) (2026)
- [EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent](https://mlsanity.com/paper/2606.17698/) (2026)
- [Evaluating Multi-Turn Bargain Skills in LLM-Based Seller Agent](https://mlsanity.com/paper/2509.06341/) (2025)
- [EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce](https://mlsanity.com/paper/2601.02752/) (2026)
- [Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition](https://mlsanity.com/paper/2604.05523/) (2026)
- [EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies](https://mlsanity.com/paper/2602.09514/) (2026)

## Built on by (most cited first, 6 of 6)

- [E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation](https://mlsanity.com/paper/2608.30730/) (2026)
- [DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum](https://mlsanity.com/paper/2609.19801/) (2026)
- [FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents](https://mlsanity.com/paper/2608.18423/) (2026)
- [FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football Management](https://mlsanity.com/paper/2609.34710/) (2026)
- [MiniCorp: The Last Mile to an AI Agent Corporation](https://mlsanity.com/paper/2610.05912/) (2026)
- [StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents](https://mlsanity.com/paper/2610.10942/) (2026)

## Abstract (arXiv)

Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3% of the mean final net assets achieved by human participants.