Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce
On the map: LLM agents and tool use → Financial trading and forecasting agents
Abstract (arXiv)
AI agents now browse and buy on people's behalf, and benchmarks measure how well the agents perform. Far less is known about the other side of the transaction: whether a merchant's website lets an agent in, publishes terms precise enough to decide a purchase, and leaves evidence of what was agreed. This paper presents a merchant-side benchmark built on those three questions and reports a pilot of it. The central measurement, decision sufficiency, rates each criterion of a concrete buyer mandate (total cost, cancellation or return terms, service date, variant identity) as determinate, indeterminate, or unknown from the merchant's own published evidence. Readiness scores carry their evidence coverage and missingness bounds; access is reported separately for a declared bot, a declared headless browser, and a user-present browser; and each observed term is mapped to the RAILS admissibility class of the evidence that could support it in a dispute. Four large merchants received the full assessment: three refused the declared profiles at their offer pages, and no mandate could be confirmed as met, because each merchant left a total or a refund term undetermined. An access census of 555 sites of merchants that two payment processors publicly name as clients found the home page unreadable to both declared profiles at 105 sites (18.9%). A declared assessment of 330 merchants, scored by deterministic rules without a rater, found that evidence coverage rather than the score decides who can be ranked: 115 merchants (35%) reached the ranking threshold, rankable profiles scored a median of 89 to 90, and material terms was the one pillar well below the others, because the all-in total was decidable at 18 merchants. The pilot shows that the measurements can be taken; it does not validate them or rank merchants, and the weights and the projected clearing band await validation against outcomes.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 62 references that are in the map. First posted on arXiv 12 Oct 2026.
Builds on
31 of 62 references.
Closest ancestors · 2
Same direction, newest first
- 2026Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer23 Sep
- 2025Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets27 Oct
Other references · 29
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
- 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
- 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
- 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 46
- 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 46
- 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 45
- 2026ClawBench: Can AI Agents Complete Everyday Online Tasks?cited by 33
- 2025ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agentscited by 28
- 2025What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commercecited by 22
17 more
- 2025Build the web for agents, not agents for the webcited by 18
- 2025WebMall -- A Multi-Shop Benchmark for Evaluating Web Agentscited by 17
- 2025Investigating the Impact of Dark Patterns on LLM-Based Web Agentscited by 17
- 2025DECEPTICON: How Dark Patterns Manipulate Web Agentscited by 15
- 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 9
- 2024Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlerscited by 7
- 2025Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical studycited by 6
- 2025MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web (Technical Report)cited by 6
- 2025Permission Manifests for Web Agentscited by 5
- 2026Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluationcited by 5
- 2026FP-Agent: Fingerprinting AI Browsing Agentscited by 3
- 2026RAILS: Verification-Native Clearing For Agentic Commercecited by 3
- 2026Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agentscited by 3
- 2026Designing Agent-Ready Websites for AI Web Agents: A Framework for Machine Readability, Actionability, and Decision Reliabilitycited by 1
- 2026terms.txt: A Consent and Compensation Protocol for Agentic Web Accesscited by 1
- 2026Signing the Transaction but Not the Decision: Whisper Attacks and a Binding Defense for AP2cited by 1
- 2026RADAR: Readiness for AI Discovery and Agentic Reachcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Agentic Commerce Bench: Measuring Fraud Detection for Agents That Spend Money
- 2026A Decision-Centered Reference Architecture for Trustworthy Agentic Commerce
- 2026MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
- 2026Agentic Commerce World: An Auditable and Verifiable Environment for Vibe Commerce
- 2026EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
- 2026Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce
- 2026MarketBench: Evaluating AI Agents as Market Participants
- 2026E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation