# User Simulation for Evaluating Information Access Systems

arXiv: https://arxiv.org/abs/2306.08550 · HTML: https://mlsanity.com/paper/2306.08550/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 86 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Evaluating Conversational Recommender Systems via User Simulation](https://mlsanity.com/paper/2006.08732/) (2020)
- [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016)
- [Analyzing and Characterizing User Intent in Information-seeking Conversations](https://mlsanity.com/paper/1804.08759/) (2018)
- [User Modeling for Task Oriented Dialogues](https://mlsanity.com/paper/1811.04369/) (2018)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [Evaluating Mixed-initiative Conversational Search Systems via User Simulation](https://mlsanity.com/paper/2204.08046/) (2022)
- [Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond](https://mlsanity.com/paper/2304.13874/) (2023)
- [MG-ShopDial: A Multi-Goal Conversational Dataset for e-Commerce](https://mlsanity.com/paper/2304.12636/) (2023)
- [QRFA: A Data-Driven Model of Information-Seeking Dialogues](https://mlsanity.com/paper/1812.10720/) (2018)
- [UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems](https://mlsanity.com/paper/2301.05544/) (2023)
- [Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems](https://mlsanity.com/paper/2205.01763/) (2022)
- [Validating Simulations of User Query Variants](https://mlsanity.com/paper/2201.07620/) (2022)

## Built on by (most cited first, 30 of 31)

- [Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions](https://mlsanity.com/paper/2312.02913/) (2023)
- [MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems](https://mlsanity.com/paper/2510.17281/) (2025)
- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [A Survey on Recent Advances in Conversational Data Generation](https://mlsanity.com/paper/2405.13003/) (2024)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluation](https://mlsanity.com/paper/2501.04410/) (2025)
- [ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders](https://mlsanity.com/paper/2602.16938/) (2026)
- [CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment](https://mlsanity.com/paper/2407.21553/) (2024)
- [Generative Information Retrieval Evaluation](https://mlsanity.com/paper/2404.08137/) (2024)
- [MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness](https://mlsanity.com/paper/2601.08118/) (2026)
- [Query Understanding in LLM-based Conversational Information Seeking](https://mlsanity.com/paper/2504.06356/) (2025)
- [Rethinking Group Recommender Systems in the Era of Generative AI: From One-Shot Recommendations to Agentic Group Decision Support](https://mlsanity.com/paper/2507.00535/) (2025)
- [Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation](https://mlsanity.com/paper/2510.16829/) (2025)
- [Verifiable User Simulation for Search and Recommendation Systems](https://mlsanity.com/paper/2606.14474/) (2026)
- [Generative Agents Navigating Digital Libraries](https://mlsanity.com/paper/2602.22529/) (2026)
- [RADIUS: Ranking, Distribution, and Significance - A Comprehensive Alignment Suite for Survey Simulation](https://mlsanity.com/paper/2603.19002/) (2026)
- [Evaluation of Agents under Simulated AI Marketplace Dynamics](https://mlsanity.com/paper/2604.14256/) (2026)
- [Disclosure-Gated User Simulation for Companion-Agent Evaluation](https://mlsanity.com/paper/2609.00982/) (2026)
- [VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity](https://mlsanity.com/paper/2501.03139/) (2025)
- [Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)](https://mlsanity.com/paper/2505.11687/) (2025)
- [JIR-Arena: The First Benchmark Dataset for Just-in-time Information Recommendation](https://mlsanity.com/paper/2505.13550/) (2025)
- [Objectifying the Subjective: Cognitive Biases in Topic Interpretations](https://mlsanity.com/paper/2507.19117/) (2025)
- [The Indispensable Role of User Simulation in the Pursuit of AGI](https://mlsanity.com/paper/2509.19456/) (2025)
- [Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems](https://mlsanity.com/paper/2604.07929/) (2026)
- [The "Curse of Knowledge" in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion](https://mlsanity.com/paper/2608.25245/) (2026)
- ["Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulators](https://mlsanity.com/paper/2608.30033/) (2026)
- [Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?](https://mlsanity.com/paper/2609.07629/) (2026)
- [Automatic multimodal UX improvement recommendations from LLM agent user simulations](https://mlsanity.com/paper/2609.22971/) (2026)
- [Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale](https://mlsanity.com/paper/2609.30137/) (2026)

## Abstract (arXiv)

Information access systems, such as search engines, recommender systems, and conversational assistants, have become integral to our daily lives as they help us satisfy our information needs. However, evaluating the effectiveness of these systems presents a long-standing and complex scientific challenge. This challenge is rooted in the difficulty of assessing a system's overall effectiveness in assisting users to complete tasks through interactive support, and further exacerbated by the substantial variation in user behaviour and preferences. To address this challenge, user simulation emerges as a promising solution. This book focuses on providing a thorough understanding of user simulation techniques designed specifically for evaluation purposes. We begin with a background of information access system evaluation and explore the diverse applications of user simulation. Subsequently, we systematically review the major research progress in user simulation, covering both general frameworks for designing user simulators, utilizing user simulation for evaluation, and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants. Realizing that user simulation is an interdisciplinary research topic, whenever possible, we attempt to establish connections with related fields, including machine learning, dialogue systems, user modeling, and economics. We end the book with a detailed discussion of important future research directions, many of which extend beyond the evaluation of information access systems and are expected to have broader impact on how to evaluate interactive intelligent systems in general.