# Verifiable User Simulation for Search and Recommendation Systems

arXiv: https://arxiv.org/abs/2606.14474 · HTML: https://mlsanity.com/paper/2606.14474/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 10 of its 17 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 10 of 10)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Cultural Bias and Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2311.14096/) (2023)
- [On Generative Agents in Recommendation](https://mlsanity.com/paper/2310.10108/) (2023)
- [User Simulation for Evaluating Information Access Systems](https://mlsanity.com/paper/2306.08550/) (2023)
- [PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation](https://mlsanity.com/paper/2506.04551/) (2025)
- [Temporal-Aware User Behaviour Simulation with Large Language Models for Recommender Systems](https://mlsanity.com/paper/2509.16895/) (2025)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)

## Built on by (most cited first, 2 of 2)

- [The "Curse of Knowledge" in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion](https://mlsanity.com/paper/2608.25245/) (2026)
- [When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark](https://mlsanity.com/paper/2610.16067/) (2026)

## Abstract (arXiv)

Large-language-model (LLM) based user simulation is increasingly adopted for evaluating search engines, recommender systems, and retrieval-augmented generation pipelines, yet most simulators remain opaque: it is difficult to determine why a simulated user made a particular choice or whether that choice is consistent with the intended user profile. Compounding this, recent research shows that LLMs can produce biased or discriminatory responses depending on user background characteristics such as language, education level, and cultural context, raising concerns about the equitable treatment of minority and disadvantaged groups. This half-day, in-person tutorial introduces a proposed design-and-audit framework that treats a user simulator as a verifiable engineering artefact composed of seven auditable components - structured Persona, task-aware Contract, matched human-vs-agent Execution, auditable Trace, persona-aligned Verification, structured Feedback, and a Refinement loop that updates personas and contracts. Through two hands-on mini-labs on recommendation-list evaluation and search-query formulation, participants will inspect simulator behaviour end-to-end, distinguish diagnostic discrepancy analysis from statistical validation, and apply checks for fidelity, credibility, and demographic bias. The tutorial targets information retrieval and recommender systems researchers and practitioners interested in user behaviour simulation and responsible AI.