# Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning

arXiv: https://arxiv.org/abs/2511.00222 · HTML: https://mlsanity.com/paper/2511.00222/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 35 of its 79 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 35)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs](https://mlsanity.com/paper/2502.09597/) (2025)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [Language Models as Agent Models](https://mlsanity.com/paper/2212.01681/) (2022)
- [Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia](https://mlsanity.com/paper/2312.03664/) (2023)
- [LLM Generated Persona is a Promise with a Catch](https://mlsanity.com/paper/2503.16527/) (2025)
- [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](https://mlsanity.com/paper/2504.18412/) (2025)
- [I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling](https://mlsanity.com/paper/2012.13391/) (2020)
- [Multi-turn Reinforcement Learning from Preference Human Feedback](https://mlsanity.com/paper/2405.14655/) (2024)
- [One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity](https://mlsanity.com/paper/2411.04427/) (2024)
- [Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores](https://mlsanity.com/paper/2403.00553/) (2024)
- [LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models](https://mlsanity.com/paper/2402.02896/) (2024)
- [Predicting Question-Answering Performance of Large Language Models through Semantic Consistency](https://mlsanity.com/paper/2311.01152/) (2023)
- [Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward](https://mlsanity.com/paper/2504.03206/) (2025)
- [Working Memory Capacity of ChatGPT: An Empirical Study](https://mlsanity.com/paper/2305.03731/) (2023)
- [Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing](https://mlsanity.com/paper/2405.07726/) (2024)
- [Know Your Audience: Do LLMs Adapt to Different Age and Education Levels?](https://mlsanity.com/paper/2312.02065/) (2023)
- [Logical Consistency of Large Language Models in Fact-checking](https://mlsanity.com/paper/2412.16100/) (2024)
- [Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning](https://mlsanity.com/paper/2310.10735/) (2023)

## Built on by (most cited first, 30 of 39)

- [HumanLM: Simulating Users with State Alignment Beats Response Imitation](https://mlsanity.com/paper/2603.03303/) (2026)
- [RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation](https://mlsanity.com/paper/2605.20204/) (2026)
- [Learning to Simulate Human Dialogue](https://mlsanity.com/paper/2601.04436/) (2026)
- [UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning](https://mlsanity.com/paper/2601.09215/) (2026)
- [The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models](https://mlsanity.com/paper/2604.24698/) (2026)
- [ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions](https://mlsanity.com/paper/2605.20087/) (2026)
- [Learning User Simulators with Turing Rewards](https://mlsanity.com/paper/2606.19336/) (2026)
- [Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors](https://mlsanity.com/paper/2605.07847/) (2026)
- [One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL](https://mlsanity.com/paper/2608.12253/) (2026)
- [ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions](https://mlsanity.com/paper/2605.24279/) (2026)
- [PTCBENCH: Benchmarking Contextual Stability of Personality Traits in LLM Systems](https://mlsanity.com/paper/2602.00016/) (2026)
- [When Can Digital Personas Reliably Approximate Human Survey Findings?](https://mlsanity.com/paper/2605.10659/) (2026)
- [Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanisms](https://mlsanity.com/paper/2605.30169/) (2026)
- [PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency](https://mlsanity.com/paper/2603.25620/) (2026)
- [The Dynamics of Delusion: Modeling Bidirectional False Belief Amplification in Human-Chatbot Dialogue](https://mlsanity.com/paper/2604.25096/) (2026)
- [Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization](https://mlsanity.com/paper/2602.11351/) (2026)
- [MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignment](https://mlsanity.com/paper/2604.13828/) (2026)
- [When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis](https://mlsanity.com/paper/2604.27228/) (2026)
- [Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experiment](https://mlsanity.com/paper/2605.21401/) (2026)
- [Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Games](https://mlsanity.com/paper/2606.17657/) (2026)
- [Quantitative Introspection in Language Models: Tracking Emotive States Across Conversation](https://mlsanity.com/paper/2603.18893/) (2026)
- [PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Users](https://mlsanity.com/paper/2605.13307/) (2026)
- [Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging](https://mlsanity.com/paper/2607.10428/) (2026)
- [MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Models](https://mlsanity.com/paper/2512.24693/) (2025)
- [Partial Policy Gradients for RL in LLMs](https://mlsanity.com/paper/2603.06138/) (2026)
- [Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility](https://mlsanity.com/paper/2605.06856/) (2026)
- [CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators](https://mlsanity.com/paper/2605.08334/) (2026)
- [Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration](https://mlsanity.com/paper/2606.06388/) (2026)
- [AGORA: Can Deliberation and Governance Gates Absorb Participation Bias in Transit Planning?](https://mlsanity.com/paper/2606.13696/) (2026)
- [Do Modules Stay in Their Lane? Role Drift in Compound LLM Systems](https://mlsanity.com/paper/2607.21627/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play. While these simulations enable scalable training and evaluation of AI agents, off-the-shelf LLMs often drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior. We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue. We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations. Using these metrics as reward signals, we apply multi-turn reinforcement learning to fine-tune LLMs for three user roles: a patient, a student, and a social chat partner. Our method reduces inconsistency by over 55%, resulting in more coherent and faithful simulated users.