User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluation
Not on the map. Why?
Abstract (arXiv)
User simulation is an emerging interdisciplinary topic with multiple critical applications in the era of Generative AI. It involves creating an intelligent agent that mimics the actions of a human user interacting with an AI system, enabling researchers to model and analyze user behaviour, generate synthetic data for training, and evaluate interactive AI systems in a controlled and reproducible manner. Because of its broad scope, research on this topic currently remains scattered across artificial intelligence, human-computer interaction, information science, computational social science, and psychology. To address this fragmented landscape of current research, this article presents a foundational synthesis. We highlight the paradigm shift from traditional predictive models to modern generative approaches, and explicitly frame critical ethical considerations -- demonstrating how controlled simulation serves not merely as a risk vector for bias, but as a powerful, proactive tool to ensure fair representation and system safety. Furthermore, we establish the theoretical connection between user simulation and the pursuit of Artificial General Intelligence, arguing that realistic simulators are indispensable catalysts for overcoming critical data and evaluation bottlenecks and optimizing personalization. Ultimately, we propose a practical, self-sustaining innovation ecosystem bridging academia and industry to advance this increasingly important technology.
Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 21 of its 54 references that are in the map. First posted on arXiv 8 Jan 2025.
Builds on
21 of 54 references.
References · 21
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
- 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
- 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
- 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
- 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
- 2023Large language models can accurately predict searcher preferencescited by 129
9 more
- 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
- 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
- 2023User Simulation for Evaluating Information Access Systemscited by 30
- 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
- 2021NeuralWOZ: Learning to Collect Task-Oriented Dialogue via Model-Based Simulationcited by 16
- 2023An Ecosystem for Personal Knowledge Graphs: A Survey and Research Roadmapcited by 11
- 2024Designing Skill-Compatible AI: Methodologies and Frameworks in Chesscited by 10
- 2024Identifying Breakdowns in Conversational Recommender Systems using User Simulationcited by 4
- 2024Report on the Workshop on Simulations for Information Access (Sim4IA 2024) at SIGIR 2024cited by 3
Built on by
Citing papers per year: 2025: 7; 2026: 9.
16 papers.
Sort:
By area:
- 2026A Survey on LLM-based Conversational User Simulationcited by 14
- 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenderscited by 10
- 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
- 2026UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learningcited by 8
- 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
- 2025Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trustcited by 6
- 2025Exploring Human-Like Thinking in Search Simulations with Large Language Modelscited by 3
- 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universecited by 3
- 2025EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactionscited by 2
- 2025Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarioscited by 1
- 2026ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisorscited by 1
- 2026DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoningcited by 0
4 more
- 2026Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulationcited by 0
- 2026KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learncited by 0
- 2026Investigating Assistant Bias in LLM User Simulators Using a Role Vectorcited by 0
- 2026Automatic multimodal UX improvement recommendations from LLM agent user simulationscited by 0
- 2026Automatic multimodal UX improvement recommendations from LLM agent user simulations19 Sep 2026
- 2026Investigating Assistant Bias in LLM User Simulators Using a Role Vector1 Sep 2026
- 2026KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn17 Aug 2026
- 2026ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisors2 Aug 2026
- 2026A Survey on LLM-based Conversational User Simulation27 Apr 2026
- 2026Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation16 Apr 2026
- 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders18 Feb 2026
- 2026DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning30 Jan 2026
- 2026UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning14 Jan 2026
- 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe22 Dec 2025
- 2025Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios30 Nov 2025
- 2025Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trust2 Jul 2025
4 more
- 2025EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions16 May 2025
- 2025Generative AI for Autonomous Driving: Frontiers and Opportunities13 May 2025
- 2025Exploring Human-Like Thinking in Search Simulations with Large Language Models10 Apr 2025
- 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning26 Feb 2025
- 2025Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trustcited by 6
- 2025Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarioscited by 1
- 2026Automatic multimodal UX improvement recommendations from LLM agent user simulationscited by 0
- 2026Automatic multimodal UX improvement recommendations from LLM agent user simulations19 Sep 2026
- 2025Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios30 Nov 2025
- 2025Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trust2 Jul 2025
- 2026A Survey on LLM-based Conversational User Simulationcited by 14
- 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenderscited by 10
- 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universecited by 3
- 2025EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactionscited by 2
- 2026ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisorscited by 1
- 2026DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoningcited by 0
- 2026Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulationcited by 0
- 2026KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learncited by 0
- 2026Investigating Assistant Bias in LLM User Simulators Using a Role Vectorcited by 0
- 2026Investigating Assistant Bias in LLM User Simulators Using a Role Vector1 Sep 2026
- 2026KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn17 Aug 2026
- 2026ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisors2 Aug 2026
- 2026A Survey on LLM-based Conversational User Simulation27 Apr 2026
- 2026Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation16 Apr 2026
- 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders18 Feb 2026
- 2026DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning30 Jan 2026
- 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe22 Dec 2025
- 2025EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions16 May 2025