mlsanityData to 14 Oct 2026

Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward

arXiv 2504.03206 ↗First posted 4 Apr 2025cs.CL cs.AIcited by 27

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Effective conversational agents like large language models (LLMs) must personalize their interactions to adapt to user preferences, personalities, and attributes across diverse domains like education and healthcare. Current methods like Reinforcement Learning from Human Feedback (RLHF), often prioritize helpfulness and safety but fall short in fostering truly empathetic, adaptive, and personalized dialogues. Existing personalization approaches typically rely on extensive user history, limiting their effectiveness for new or context-limited users. To address these limitations, we propose leveraging a user model to incorporate a curiosity-based intrinsic reward into multi-turn RLHF. This novel reward mechanism encourages the LLM agent to actively infer user traits by optimizing conversations to improve its user model's accuracy. Consequently, the agent delivers more personalized interactions by learning more about the user. We demonstrate our method's effectiveness in two distinct domains: significantly improving personalization performance in a conversational recommendation task, and personalizing conversations for different learning styles in an educational setting. We show improved generalization capabilities compared to traditional multi-turn RLHF, all while maintaining conversation quality. Our method offers a promising solution for creating more personalized, adaptive, and engaging conversational agents.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 49 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

19 of 49 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  7. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  8. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  9. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  10. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  11. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  12. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
7 more
  1. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  2. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  3. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 67
  4. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 51
  5. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  6. 2025Language Model Personalization via Reward Factorizationcited by 31
  7. 2024RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 19.

27 papers.

Sort:

By area:

  1. 2025Self-Questioning Language Modelscited by 50
  2. 2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 39
  3. 2026A Survey on LLM-based Conversational User Simulationcited by 18
  4. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 14
  5. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agentscited by 9
  6. 2026Learning to Simulate Human Dialoguecited by 9
  7. 2026MultiSessionCollab: Learning User Preferences with Memory to Improve Long-Term Collaborationcited by 7
  8. 2026On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agentscited by 6
  9. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 6
  10. 2025Reducing Belief Deviation in Reinforcement Learning for Active Reasoningcited by 4
  11. 2026Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interactioncited by 4
  12. 2025Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interactioncited by 3
15 more
  1. 2026TRUST-SQL: Tool-Integrated Multi-Turn Reinforcement Learning for Text-to-SQL over Unknown Schemascited by 3
  2. 2026SHAPE: Unifying Safety, Helpfulness and Pedagogy for Educational LLMscited by 3
  3. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
  4. 2025Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgmentcited by 1
  5. 2026Dialogue Model Optimization via Agent Game and Adaptive Tree-based GRPOcited by 1
  6. 2026Unlocking Proactivity in Task-Oriented Dialoguecited by 1
  7. 2026Hypotheses-Guided Self Distillation for Continual Personalizationcited by 1
  8. 2026Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learningcited by 1
  9. 2025WHEN TO ACT, WHEN TO WAIT: Modeling the Intent-Action Alignment Problem in Dialoguecited by 0
  10. 2025Offline Policy Evaluation of Multi-Turn LLM Health Coaching with Real Userscited by 0
  11. 2026Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Oncecited by 0
  12. 2026Communication Policy Evolution for Proactive LLM Agentscited by 0
  13. 2026Ask or Answer: A Decision Framework for Multi-Turn Health Misinformation Interventioncited by 0
  14. 2026Training User Simulators with Latent Intrinsic Feedbackcited by 0
  15. 2026PersonaFactor: Preference Factors with Dynamic Selection for LLM Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Era of Real-World Human Interaction: RL from User Conversations
  2. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment
  3. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  4. 2025From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational System
  5. 2026Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering
  6. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training
  7. 2026Ask to Be Sure: Informative Interactions for Confident Multi-Turn LLM Recommendation
  8. 2025Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager