# Learning User Simulators with Turing Rewards

arXiv: https://arxiv.org/abs/2606.19336 · HTML: https://mlsanity.com/paper/2606.19336/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 64 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 30 of 46)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [Aligning Language Models to User Opinions](https://mlsanity.com/paper/2305.14929/) (2023)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [ConvoKit: A Toolkit for the Analysis of Conversations](https://mlsanity.com/paper/2005.04246/) (2020)
- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2511.00222/) (2025)
- [HumanLM: Simulating Users with State Alignment Beats Response Imitation](https://mlsanity.com/paper/2603.03303/) (2026)
- [Can Language Models Reason about Individualistic Human Values and Preferences?](https://mlsanity.com/paper/2410.03868/) (2024)
- [SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs](https://mlsanity.com/paper/2506.05598/) (2025)
- [Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data](https://mlsanity.com/paper/2503.20749/) (2025)
- [Cognitive Foundations for Reasoning and Their Manifestation in LLMs](https://mlsanity.com/paper/2511.16660/) (2025)

## Built on by (most cited first, 7 of 7)

- [Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation](https://mlsanity.com/paper/2608.09420/) (2026)
- [Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation](https://mlsanity.com/paper/2607.27816/) (2026)
- [CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking](https://mlsanity.com/paper/2610.02460/) (2026)
- [Dataset Signatures in Human-LLM Interactions and User Modeling](https://mlsanity.com/paper/2610.05534/) (2026)
- [MIMESIS: Learning User Simulators as Training Environments for Interactive Agents](https://mlsanity.com/paper/2610.09484/) (2026)
- [Training User Simulators with Latent Intrinsic Feedback](https://mlsanity.com/paper/2610.13617/) (2026)
- [Cowork-Gym: Synthesizing Interactive Work Environments at Scale](https://mlsanity.com/paper/2610.17670/) (2026)

## Abstract (arXiv)

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.