mlsanityData to 14 Oct 2026

LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?

arXiv 2509.21501 ↗First posted 25 Sep 2025cs.HC cs.CLcited by 15

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Agentic AI is emerging, capable of executing tasks through natural language, such as Copilot for coding or Amazon Rufus for shopping. Evaluating these systems is challenging, as their rapid evolution outpaces traditional human evaluation. Researchers have proposed LLM Agents to simulate participants as digital twins, but it remains unclear to what extent a digital twin can represent a specific customer in multi-turn interaction with an agentic AI system. In this paper, we recruited 40 human participants to shop with Amazon Rufus, collected their personas, interaction traces, and UX feedback, and then created digital twins to repeat the task. Pairwise comparison of human and digital-twin traces shows that while agents often explored more diverse choices, their action patterns aligned with humans and yielded similar design feedback. This study is the first to quantify how closely LLM agents can mirror human multi-turn interaction with an agentic AI system, highlighting their potential for scalable evaluation.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 24 of its 117 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

24 of 117 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data26 Mar
  2. 2024Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View23 May

Other references · 22

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  3. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
  4. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  5. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  6. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 76
  7. 2024Design Principles for Generative AI Applicationscited by 57
  8. 2019What Makes a Good Conversation? Challenges in Designing Truly Conversational Agentscited by 42
  9. 2025UXAgent: An LLM Agent-Based Usability Testing Framework for Web Designcited by 40
  10. 2023AXNav: Replaying Accessibility Tests from Natural Languagecited by 31
  11. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 30
  12. 2025OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulationcited by 28
10 more
  1. 2024Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seekingcited by 26
  2. 2025AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agentscited by 19
  3. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learningcited by 19
  4. 2024Objection Overruled! Lay People can Distinguish Large Language Models from Lawyers, but still Favour Advice from an LLMcited by 16
  5. 2025Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agentscited by 11
  6. 2024Comparing Traditional and LLM-based Search for Image Geolocationcited by 10
  7. 2023Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)cited by 6
  8. 2025User Experience with LLM-powered Conversational Recommendation Systems: A Case of Music Recommendationcited by 6
  9. 2024Understanding How Psychological Distance Influences User Preferences in Conversational Versus Web Searchcited by 4
  10. 2021Agents for Automated User Experience Testingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2026ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistantscited by 10
  2. 2025SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluationcited by 6
  3. 2026SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commercecited by 4
  4. 2025Hybrid Agentic AI and Multi-Agent Systems in Smart Manufacturingcited by 2
  5. 2026VISTA: A Versatile Interactive User Simulation Toolkit for Agent Evaluationcited by 2
  6. 2026TruthTensor: Evaluating LLMs through Human Imitation on Prediction Market under Drift and Holistic Reasoningcited by 1
  7. 2026Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistantscited by 1
  8. 2026SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agentscited by 1
  9. 2026Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commercecited by 1
  10. 2026Autonoma: A Hierarchical Multi-Agent Framework for End-to-End Workflow Automationcited by 0
  11. 2026SHM-Agents: A Generalist-Specialist Integrated Agent System for Structural Health Monitoringcited by 0
  12. 2026From Triggers to Emotions: A CPM-Grounded Appraisal Multi-Agent for Dynamic Emotional Evolution in Persona-Based Dialoguecited by 0
3 more
  1. 2026Agentic Share-of-Search: A Multi-Agent AI System for Competitive Decision-Making in LLM-Mediated E-Commercecited by 0
  2. 2026Automatic multimodal UX improvement recommendations from LLM agent user simulationscited by 0
  3. 2026SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Agentic Large Language Models, a survey
  2. 2026When AI Designs AI: Innovation or Imitation?
  3. 2025FaMA: LLM-Empowered Agentic Assistant for Consumer-to-Consumer Marketplace
  4. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROI
  5. 2025PAARS: Persona Aligned Agentic Retail Shoppers
  6. 2026From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agents
  7. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
  8. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations