mlsanityData to 14 Oct 2026

Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy

arXiv 2505.21907 ↗First posted 28 May 2025cs.AI cs.CL cs.HCcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

AI copilots represent a new generation of AI-powered systems designed to assist users, particularly knowledge workers and developers, in complex, context-rich tasks. As these systems become more embedded in daily workflows, personalization has emerged as a critical factor for improving usability, effectiveness, and user satisfaction. Central to this personalization is preference optimization: the system's ability to detect, interpret, and align with individual user preferences. While prior work in intelligent assistants and optimization algorithms is extensive, their intersection within AI copilots remains underexplored. This survey addresses that gap by examining how user preferences are operationalized in AI copilots. We investigate how preference signals are sourced, modeled across different interaction stages, and refined through feedback loops. Building on a comprehensive literature review, we define the concept of an AI copilot and introduce a taxonomy of preference optimization techniques across pre-, mid-, and post-interaction phases. Each technique is evaluated in terms of advantages, limitations, and design implications. By consolidating fragmented efforts across AI personalization, human-AI interaction, and language model adaptation, this work offers both a unified conceptual foundation and a practical design perspective for building user-aligned, persona-aware AI copilots that support end-to-end adaptability and deployment.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 72 of its 130 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

72 of 130 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Aligning LLMs with Individual Preferences via Interaction4 Oct
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference7 Mar
  3. 2023Pretraining Language Models with Human Preferences16 Feb
  4. 2020Learning to summarize from human feedback2 Sep

Other references · 68

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2024A Survey on LLM-as-a-Judgecited by 1,423
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
38 more
  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  2. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  3. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  5. 2024Iterative Reasoning Preference Optimizationcited by 246
  6. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  7. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  8. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 175
  9. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  10. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  11. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 127
  12. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  13. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  14. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  15. 2019Personalizing Dialogue Agents via Meta-Learningcited by 83
  16. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 78
  17. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  18. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 74
  19. 2022Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Supportcited by 68
  20. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  21. 2021A Survey on Accuracy-oriented Neural Recommendation: From Collaborative Filtering to Information-rich Recommendationcited by 59
  22. 2024Active Preference Learning for Large Language Modelscited by 58
  23. 2020Towards Persona-Based Empathetic Conversational Modelscited by 53
  24. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  25. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 48
  26. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 44
  27. 2021The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systemscited by 39
  28. 2024Bayesian Preference Elicitation with Language Modelscited by 34
  29. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  30. 2023Active Preference Inference using Language Models and Probabilistic Reasoningcited by 29
  31. 2024User Modeling and User Profiling: A Comprehensive Surveycited by 29
  32. 2022A Literature Survey of Recent Advances in Chatbotscited by 26
  33. 2022A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generationcited by 25
  34. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  35. 2022Personalized Dialogue Generation with Persona-Adaptive Attentioncited by 24
  36. 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 22
  37. 2024Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluationscited by 20
  38. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026PersoPilot: An Adaptive AI-Copilot for Transparent Contextualized Persona Classification and Personalized Response Generationcited by 1
  2. 2026A New Strategy for Artificial Intelligence: Training Foundation Models Directly on Human Brain Datacited by 0
  3. 2026PersoDPO: Scalable Preference Optimization for Instruction-Adherent, Persona-Grounded Dialogue via Multi-LLM Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PREDICT: Preference Reasoning by Evaluating Decomposed preferences Inferred from Candidate Trajectories
  2. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interaction
  3. 2026MAESTRO: Adapting GUIs and Guiding Navigation with User Preferences in Conversational Agents with GUIs ↗
  4. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Models
  5. 2025ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation
  6. 2026Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions
  7. 2026AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
  8. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users