mlsanityData to 7 Oct 2026

Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation

arXiv 2403.09738 ↗First posted 13 Mar 2024cs.CL cs.AI cs.IRcited by 45

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Synthetic users are cost-effective proxies for real users in the evaluation of conversational recommender systems. Large language models show promise in simulating human-like behavior, raising the question of their ability to represent a diverse population of users. We introduce a new protocol to measure the degree to which language models can accurately emulate human behavior in conversational recommendation. This protocol is comprised of five tasks, each designed to evaluate a key property that a synthetic user should exhibit: choosing which items to talk about, expressing binary preferences, expressing open-ended preferences, requesting recommendations, and giving feedback. Through evaluation of baseline simulators, we demonstrate these tasks effectively reveal deviations of language models from human behavior, and offer insights on how to reduce the deviations with model selection and prompting strategies.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

23 of 38 references.

References · 23

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  5. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  6. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  7. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 228
  8. 2018Towards Deep Conversational Recommendationscited by 174
  9. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 126
  10. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  11. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  12. 2018Conversational Recommender Systemcited by 115
11 more
  1. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 113
  2. 2023GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretationcited by 101
  3. 2020Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systemscited by 98
  4. 2023PALR: Personalization Aware LLMs for Recommendationcited by 96
  5. 2018Dialog-based Interactive Image Retrievalcited by 84
  6. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
  7. 2020Interactive Path Reasoning on Graph for Conversational Recommendationcited by 72
  8. 2020Evaluating Conversational Recommender Systems via User Simulationcited by 46
  9. 2023A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understandingcited by 31
  10. 2023Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyondcited by 23
  11. 2023An In-depth Investigation of User Response Simulation for Conversational Searchcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 17; 2026: 25.

45 papers.

Sort:

By area:

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  2. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  3. 2025A Survey on LLM-powered Agents for Recommender Systemscited by 49
  4. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 44
  5. 2025Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profilescited by 38
  6. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 36
  7. 2025SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluationcited by 32
  8. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
  9. 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systemscited by 18
  10. 2025Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human Statescited by 17
  11. 2025CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactionscited by 16
  12. 2025PAARS: Persona Aligned Agentic Retail Shopperscited by 14
33 more
  1. 2026HorizonBench: Long-Horizon Personalization with Evolving Preferencescited by 14
  2. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  3. 2024ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questionscited by 10
  4. 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenderscited by 10
  5. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMscited by 7
  6. 2026AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluationcited by 6
  7. 2026Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Tracescited by 6
  8. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  9. 2025Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated Usercited by 5
  10. 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agentscited by 5
  11. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  12. 2026Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluationcited by 3
  13. 2026Autonomous Information Seeking: A Roadmap for Agentic Recommender Systemscited by 3
  14. 2025AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agentscited by 2
  15. 2026Multi-Agent Home Energy Management Assistantcited by 2
  16. 2025InterFeedback: Unveiling Interactive Intelligence of Large Multimodal Models via Human Feedbackcited by 1
  17. 2025Explainable Recommendation with Simulated Human Feedbackcited by 1
  18. 2025Prompt Optimization Across Multiple Agents for Representing Diverse Human Populationscited by 1
  19. 2026Entropy Guided Diversification and Preference Elicitation in Agentic Recommendation Systemscited by 1
  20. 2026ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisorscited by 1
  21. 2025LatentCRS: A Variational EM Framework for Bridging Semantics and Behavior in LLM-based Conversational Recommendationcited by 0
  22. 2025From What to Respond to When to Respond: Timely Response Generation for Open-domain Dialogue Agentscited by 0
  23. 2026PersoDPO: Scalable Preference Optimization for Instruction-Adherent, Persona-Grounded Dialogue via Multi-LLM Evaluationcited by 0
  24. 2026DALI: LLM-Agent Enhanced Dual-Stream Adaptive Leadership Identification for Group Recommendationscited by 0
  25. 2026User Simulator-Guided Multi-Turn Preference Optimization for Reasoning LLM-based Conversational Recommendationcited by 0
  26. 2026Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systemscited by 0
  27. 2026Bridging Textual Profiles and Latent User Embeddings for Personalizationcited by 0
  28. 2026LLM Consumer Behavior Theory: Foundations of a Novel Research Fieldcited by 0
  29. 2026Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Frameworkcited by 0
  30. 2026SR-Agent: An Experience-Driven Agentic Framework for Post-Ranking Strategy Refinement in E-Commerce Recommendationcited by 0
  31. 2026"Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulatorscited by 0
  32. 2026Rhythms of Work: Multi-Scale Interpretation of Human Behavioral Traces for Workplace Agentscited by 0
  33. 2026Re2A: Situated Conversational Recommendation via Rubric-based Preference Reasoning and Alignmentcited by 0