mlsanityData to 14 Oct 2026

Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning

arXiv 2511.00222 ↗First posted 31 Oct 2025cs.CL cs.AIcited by 39

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play. While these simulations enable scalable training and evaluation of AI agents, off-the-shelf LLMs often drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior. We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue. We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations. Using these metrics as reward signals, we apply multi-turn reinforcement learning to fine-tune LLMs for three user roles: a patient, a student, and a social chat partner. Our method reduces inconsistency by over 55%, resulting in more coherent and faithful simulated users.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 79 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

35 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Multi-turn Reinforcement Learning from Preference Human Feedback23 May

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  7. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  11. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  12. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 209
22 more
  1. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  2. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  3. 2022Language Models as Agent Modelscited by 132
  4. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 126
  5. 2025LLM Generated Persona is a Promise with a Catchcited by 81
  6. 2025Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providerscited by 69
  7. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  8. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 58
  9. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  10. 2024LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Modelscited by 33
  11. 2023Predicting Question-Answering Performance of Large Language Models through Semantic Consistencycited by 28
  12. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Rewardcited by 27
  13. 2023Working Memory Capacity of ChatGPT: An Empirical Studycited by 23
  14. 2024Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playingcited by 17
  15. 2023Know Your Audience: Do LLMs Adapt to Different Age and Education Levels?cited by 16
  16. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  17. 2023Building Persona Consistent Dialogue Agents with Offline Reinforcement Learningcited by 14
  18. 2024A review on the use of large language models as virtual tutorscited by 13
  19. 2024IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interactioncited by 12
  20. 2024Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Modelscited by 12
  21. 2024CharacterGPT: A Persona Reconstruction Framework for Role-Playing Agentscited by 10
  22. 2023On Sarcasm Detection with OpenAI GPT-based Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 38.

39 papers.

Sort:

By area:

  1. 2026HumanLM: Simulating Users with State Alignment Beats Response Imitationcited by 37
  2. 2026RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulationcited by 12
  3. 2026Learning to Simulate Human Dialoguecited by 9
  4. 2026UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learningcited by 9
  5. 2026The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Modelscited by 8
  6. 2026ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactionscited by 7
  7. 2026Learning User Simulators with Turing Rewardscited by 7
  8. 2026Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviorscited by 6
  9. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 6
  10. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  11. 2026PTCBENCH: Benchmarking Contextual Stability of Personality Traits in LLM Systemscited by 4
  12. 2026When Can Digital Personas Reliably Approximate Human Survey Findings?cited by 4
27 more
  1. 2026Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanismscited by 4
  2. 2026PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistencycited by 3
  3. 2026The Dynamics of Delusion: Modeling Bidirectional False Belief Amplification in Human-Chatbot Dialoguecited by 3
  4. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
  5. 2026MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignmentcited by 2
  6. 2026When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysiscited by 2
  7. 2026Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experimentcited by 2
  8. 2026Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Gamescited by 2
  9. 2026Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationcited by 1
  10. 2026PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Userscited by 1
  11. 2026Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judgingcited by 1
  12. 2025MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Modelscited by 0
  13. 2026Partial Policy Gradients for RL in LLMscited by 0
  14. 2026Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utilitycited by 0
  15. 2026CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulatorscited by 0
  16. 2026Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaborationcited by 0
  17. 2026AGORA: Can Deliberation and Governance Gates Absorb Participation Bias in Transit Planning?cited by 0
  18. 2026Do Modules Stay in Their Lane? Role Drift in Compound LLM Systemscited by 0
  19. 2026SteerDuplex: Steerable Duplex Speech Dialogue Modelscited by 0
  20. 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0
  21. 2026PersMem: Internalizing Personality into Dual-Pathway Memory for LLM Agentscited by 0
  22. 2026OSPD: On-Policy Self-Distillation for Persona-Consistent Dialoguecited by 0
  23. 2026PersonaManifold: Revealing and Exploiting Curved Geometry in LLM Persona Representationscited by 0
  24. 2026Proactive Dialogue Policy Optimization via Cognitive-State Transitioncited by 0
  25. 2026Benchmarking Psychological Dynamics in Generative Agentscited by 0
  26. 2026Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulationcited by 0
  27. 2026Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SPASM: Stable Persona-driven Agent Simulation for Multi-turn Dialogue Generation
  2. 2026Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialogue
  3. 2025Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning
  4. 2026Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
  5. 2024Examining Identity Drift in Conversations of LLM Agents
  6. 2025TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulation
  7. 2025Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles
  8. 2026PB-GRPO: Learning Socially Adaptive LLM Agents from Persona-Driven Simulation with Preference-Batched GRPO