# Simulating Before Planning: Constructing Intrinsic User World Model for User-Tailored Dialogue Policy Planning arXiv: https://arxiv.org/abs/2504.13643 ยท HTML: https://mlsanity.com/paper/2504.13643/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in World models for language agents, one of 11 directions in World models. - Builds on 34 of its 53 references that are in the map. - First posted on arXiv 18 Apr 2025. ## Placed in - World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/) ## Builds on (most cited first, 30 of 34) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [World Models](https://mlsanity.com/paper/1803.10122/) (2018) - [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023) - [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023) - [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019) - [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021) - [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024) - [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019) - [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023) - [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024) - [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018) - [Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://mlsanity.com/paper/2305.13626/) (2023) - [Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://mlsanity.com/paper/2401.12474/) (2024) - [Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents](https://mlsanity.com/paper/2311.00262/) (2023) - [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023) - [Is ChatGPT Equipped with Emotional Dialogue Capabilities?](https://mlsanity.com/paper/2304.09582/) (2023) - [The Oscars of AI Theater: A Survey on Role-Playing with Language Models](https://mlsanity.com/paper/2407.11484/) (2024) - [Evaluating Conversational Recommender Systems via User Simulation](https://mlsanity.com/paper/2006.08732/) (2020) - [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016) - [Planning Like Human: A Dual-process Framework for Dialogue Planning](https://mlsanity.com/paper/2406.05374/) (2024) - [Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Models](https://mlsanity.com/paper/2305.17878/) (2023) - [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022) - [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024) - [SweetieChat: A Strategy-Enhanced Role-playing Framework for Diverse Scenarios Handling Emotional Support Agent](https://mlsanity.com/paper/2412.08389/) (2024) - [GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers](https://mlsanity.com/paper/2208.10817/) (2022) - [Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulation](https://mlsanity.com/paper/2403.06769/) (2024) - [Dialogue Planning via Brownian Bridge Stochastic Process for Goal-directed Proactive Dialogue](https://mlsanity.com/paper/2305.05290/) (2023) - [Simulation-Free Hierarchical Latent Policy Planning for Proactive Dialogues](https://mlsanity.com/paper/2412.14584/) (2024) ## Built on by (most cited first, 7 of 7) - [PRINCIPLES: Synthetic Strategy Memory for Proactive Dialogue Agents](https://mlsanity.com/paper/2509.17459/) (2025) - [DialogXpert: Driving Intelligent and Emotion-Aware Conversations through Online Value-Based Reinforcement Learning with LLM Priors](https://mlsanity.com/paper/2505.17795/) (2025) - [A General Highly Accurate Online Planning Method Integrating Large Language Models into Nested Rollout Policy Adaptation for Dialogue Tasks](https://mlsanity.com/paper/2511.21706/) (2025) - [Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies](https://mlsanity.com/paper/2602.22696/) (2026) - [Modeling Multiple Support Strategies within a Single Turn for Emotional Support Conversations](https://mlsanity.com/paper/2604.17972/) (2026) - [UP-NRPA: User Portrait based Nested Rollout Policy Adaptation for Planning with Large Language Models in Goal-oriented Dialogue Systems](https://mlsanity.com/paper/2606.13683/) (2026) - [ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation](https://mlsanity.com/paper/2609.11101/) (2026) ## Abstract (arXiv) Recent advancements in dialogue policy planning have emphasized optimizing system agent policies to achieve predefined goals, focusing on strategy design, trajectory acquisition, and efficient training paradigms. However, these approaches often overlook the critical role of user characteristics, which are essential in real-world scenarios like conversational search and recommendation, where interactions must adapt to individual user traits such as personality, preferences, and goals. To address this gap, we first conduct a comprehensive study utilizing task-specific user personas to systematically assess dialogue policy planning under diverse user behaviors. By leveraging realistic user profiles for different tasks, our study reveals significant limitations in existing approaches, highlighting the need for user-tailored dialogue policy planning. Building on this foundation, we present the User-Tailored Dialogue Policy Planning (UDP) framework, which incorporates an Intrinsic User World Model to model user traits and feedback. UDP operates in three stages: (1) User Persona Portraying, using a diffusion model to dynamically infer user profiles; (2) User Feedback Anticipating, leveraging a Brownian Bridge-inspired anticipator to predict user reactions; and (3) User-Tailored Policy Planning, integrating these insights to optimize response strategies. To ensure robust performance, we further propose an active learning approach that prioritizes challenging user personas during training. Comprehensive experiments on benchmarks, including collaborative and non-collaborative settings, demonstrate the effectiveness of UDP in learning user-specific dialogue strategies. Results validate the protocol's utility and highlight UDP's robustness, adaptability, and potential to advance user-centric dialogue systems.