# LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants? arXiv: https://arxiv.org/abs/2509.21501 ยท HTML: https://mlsanity.com/paper/2509.21501/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. - Builds on 24 of its 117 references that are in the map. - First posted on arXiv 25 Sep 2025. ## Placed in - LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/) ## Builds on (most cited first, 24 of 24) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023) - [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021) - [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024) - [Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents](https://mlsanity.com/paper/2405.02957/) (2024) - [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://mlsanity.com/paper/2506.21506/) (2025) - [Design Principles for Generative AI Applications](https://mlsanity.com/paper/2401.14484/) (2024) - [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019) - [UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design](https://mlsanity.com/paper/2502.12561/) (2025) - [Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View](https://mlsanity.com/paper/2405.14744/) (2024) - [AXNav: Replaying Accessibility Tests from Natural Language](https://mlsanity.com/paper/2310.02424/) (2023) - [Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation](https://mlsanity.com/paper/2507.21028/) (2025) - [Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data](https://mlsanity.com/paper/2503.20749/) (2025) - [OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation](https://mlsanity.com/paper/2506.05606/) (2025) - [Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking](https://mlsanity.com/paper/2402.08030/) (2024) - [AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents](https://mlsanity.com/paper/2504.09723/) (2025) - [Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning](https://mlsanity.com/paper/2507.17842/) (2025) - [Objection Overruled! Lay People can Distinguish Large Language Models from Lawyers, but still Favour Advice from an LLM](https://mlsanity.com/paper/2409.07871/) (2024) - [Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents](https://mlsanity.com/paper/2504.17934/) (2025) - [Comparing Traditional and LLM-based Search for Image Geolocation](https://mlsanity.com/paper/2401.10184/) (2024) - [Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)](https://mlsanity.com/paper/2303.03638/) (2023) - [User Experience with LLM-powered Conversational Recommendation Systems: A Case of Music Recommendation](https://mlsanity.com/paper/2502.15229/) (2025) - [Understanding How Psychological Distance Influences User Preferences in Conversational Versus Web Search](https://mlsanity.com/paper/2409.19982/) (2024) - [Agents for Automated User Experience Testing](https://mlsanity.com/paper/2104.06220/) (2021) ## Built on by (most cited first, 15 of 15) - [ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants](https://mlsanity.com/paper/2601.18225/) (2026) - [SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation](https://mlsanity.com/paper/2510.11997/) (2025) - [SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce](https://mlsanity.com/paper/2602.01443/) (2026) - [Hybrid Agentic AI and Multi-Agent Systems in Smart Manufacturing](https://mlsanity.com/paper/2511.18258/) (2025) - [VISTA: A Versatile Interactive User Simulation Toolkit for Agent Evaluation](https://mlsanity.com/paper/2606.11079/) (2026) - [TruthTensor: Evaluating LLMs through Human Imitation on Prediction Market under Drift and Holistic Reasoning](https://mlsanity.com/paper/2601.13545/) (2026) - [Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants](https://mlsanity.com/paper/2603.03565/) (2026) - [SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents](https://mlsanity.com/paper/2605.19219/) (2026) - [Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce](https://mlsanity.com/paper/2606.12924/) (2026) - [Autonoma: A Hierarchical Multi-Agent Framework for End-to-End Workflow Automation](https://mlsanity.com/paper/2603.19270/) (2026) - [SHM-Agents: A Generalist-Specialist Integrated Agent System for Structural Health Monitoring](https://mlsanity.com/paper/2605.12916/) (2026) - [From Triggers to Emotions: A CPM-Grounded Appraisal Multi-Agent for Dynamic Emotional Evolution in Persona-Based Dialogue](https://mlsanity.com/paper/2607.07824/) (2026) - [Agentic Share-of-Search: A Multi-Agent AI System for Competitive Decision-Making in LLM-Mediated E-Commerce](https://mlsanity.com/paper/2609.11190/) (2026) - [Automatic multimodal UX improvement recommendations from LLM agent user simulations](https://mlsanity.com/paper/2609.22971/) (2026) - [SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling](https://mlsanity.com/paper/2609.38397/) (2026) ## Abstract (arXiv) Agentic AI is emerging, capable of executing tasks through natural language, such as Copilot for coding or Amazon Rufus for shopping. Evaluating these systems is challenging, as their rapid evolution outpaces traditional human evaluation. Researchers have proposed LLM Agents to simulate participants as digital twins, but it remains unclear to what extent a digital twin can represent a specific customer in multi-turn interaction with an agentic AI system. In this paper, we recruited 40 human participants to shop with Amazon Rufus, collected their personas, interaction traces, and UX feedback, and then created digital twins to repeat the task. Pairwise comparison of human and digital-twin traces shows that while agents often explored more diverse choices, their action patterns aligned with humans and yielded similar design feedback. This study is the first to quantify how closely LLM agents can mirror human multi-turn interaction with an agentic AI system, highlighting their potential for scalable evaluation.