# Training User Simulators with Latent Intrinsic Feedback

arXiv: https://arxiv.org/abs/2610.13617 · HTML: https://mlsanity.com/paper/2610.13617/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 52 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 30 of 44)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks](https://mlsanity.com/paper/2503.15478/) (2025)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [A User Simulator for Task-Completion Dialogues](https://mlsanity.com/paper/1612.05688/) (2016)
- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?](https://mlsanity.com/paper/2510.05444/) (2025)
- [How to Build User Simulators to Train RL-based Dialog Systems](https://mlsanity.com/paper/1909.01388/) (2019)
- [Training Proactive and Personalized LLM Agents](https://mlsanity.com/paper/2511.02208/) (2025)
- [SWE-chat: Coding Agent Interactions From Real Users in the Wild](https://mlsanity.com/paper/2604.20779/) (2026)
- [Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles](https://mlsanity.com/paper/2502.18968/) (2025)
- [Finetuning LLMs for Human Behavior Prediction in Social Science Experiments](https://mlsanity.com/paper/2509.05830/) (2025)
- [HumanLM: Simulating Users with State Alignment Beats Response Imitation](https://mlsanity.com/paper/2603.03303/) (2026)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [ChatBench: From Static Benchmarks to Human-AI Evaluation](https://mlsanity.com/paper/2504.07114/) (2025)
- [UserRL: Training Interactive User-Centric Agent via Reinforcement Learning](https://mlsanity.com/paper/2509.19736/) (2025)
- [Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data](https://mlsanity.com/paper/2503.20749/) (2025)
- [Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward](https://mlsanity.com/paper/2504.03206/) (2025)
- [Goal Alignment in LLM-Based User Simulators for Conversational AI](https://mlsanity.com/paper/2507.20152/) (2025)
- [IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering](https://mlsanity.com/paper/2408.13545/) (2024)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systems](https://mlsanity.com/paper/2405.08035/) (2024)
- [OdysSim: Building Foundation Models for Human Behavior Simulation](https://mlsanity.com/paper/2606.14199/) (2026)
- [ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders](https://mlsanity.com/paper/2602.16938/) (2026)

## Abstract (arXiv)

User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation has emphasized generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such evaluation signals may not be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, namely, models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversation turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that intrinsic feedback improves simulation realism and critiquing behaviors versus prompted simulator baselines, while also providing turn-level evaluations that existing fine-tuned simulators lack. Further, evaluations generated by our trained simulators correlate better with human judgments than those of prompted simulators of the same size. Lastly, our experiments reveal that simulators with intrinsic feedback provide effective reward signals for multi-turn RL training, matching setups that rely on external judge LLMs for rewards while cutting RL compute by more than half.