mlsanityData to 7 Oct 2026

A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models

arXiv 2504.07070 ↗First posted 9 Apr 2025cs.CLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Personalized preference alignment for large language models (LLMs), the process of tailoring LLMs to individual users' preferences, is an emerging research direction spanning the area of NLP and personalization. In this survey, we present an analysis of works on personalized alignment and modeling for LLMs. We introduce a taxonomy of preference alignment techniques, including training time, inference time, and additionally, user-modeling based methods. We provide analysis and discussion on the strengths and limitations of each group of techniques and then cover evaluation, benchmarks, as well as open problems in the field.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 88 of its 108 references that are in the map. First posted on arXiv 9 Apr 2025.

Builds on

88 of 108 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas20 Jan
  2. 2024ComPO: Community Preferences for Language Model Personalization21 Oct
  3. 2024Aligning LLMs with Individual Preferences via Interaction4 Oct
  4. 2024PersonalLLM: Tailoring LLMs to Individual Preferences30 Sep
  5. 2024Improving Context-Aware Preference Modeling for Language Models20 Jul
  6. 2024Orchestrating LLMs with Different Personalizations4 Jul
  7. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization28 May
  8. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  9. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  10. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  11. 2023Learning Personalized Alignment for Evaluating Open-ended Text Generation5 Oct
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec

Other references · 76

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2023Whose Opinions Do Language Models Reflect?cited by 759
  9. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  10. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  11. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  12. 2023A Survey on Large Language Models for Recommendationcited by 378
38 more
  1. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  4. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  5. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  6. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  7. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  8. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  9. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  10. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  11. 2023Controlled Decoding from Language Modelscited by 133
  12. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  13. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 121
  14. 2022A Survey on the Fairness of Recommender Systemscited by 102
  15. 2024Personalization of Large Language Models: A Surveycited by 100
  16. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  17. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  18. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  19. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 91
  20. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  21. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  22. 2024Tuning Language Models by Proxycited by 88
  23. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  24. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  25. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  26. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  27. 2024Decoding-time Realignment of Language Modelscited by 67
  28. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  29. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  30. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 59
  31. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 59
  32. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  33. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  34. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  35. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  36. 2023Teach LLMs to Personalize -- An Approach inspired by Writing Educationcited by 48
  37. 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 47
  38. 2023LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordinationcited by 46

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 18.

25 papers.

Sort:

By area:

  1. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  2. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignmentcited by 22
  3. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  4. 2025In-context Ranking Preference Optimizationcited by 9
  5. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  6. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 5
  7. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inferencecited by 5
  8. 2026PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harmcited by 5
  9. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  10. 2026PERSPECTRA: A Scalable and Configurable Pluralist Benchmark of Perspectives from Argumentscited by 3
  11. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  12. 2026Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answeringcited by 2
13 more
  1. 2026Preference-Aware Rubric Learning for Personalized Evaluationcited by 2
  2. 2026APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMscited by 1
  3. 2026Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneckcited by 1
  4. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1
  5. 2026Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routingcited by 0
  6. 2026When to Ask a Question: Understanding Communication Strategies in Generative AI Toolscited by 0
  7. 2026An Interactive Paradigm for Deep Researchcited by 0
  8. 2026A Roadmap to Impactful Pluralistic Alignment Researchcited by 0
  9. 2026Where Models Converge and Humans Diverge: A Coverage Framework for Distributional Pluralism in Open-Ended Generationcited by 0
  10. 2026Cross-Model Humor Preference Modeling with Cards Against Humanitycited by 0
  11. 2026Procedural Fairness Failures in RLHF from Preference Averagingcited by 0
  12. 2026Bounded Personas Match Retrieval on Classification but Not Regression for a Frozen Agentcited by 0
  13. 2026Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applicationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  3. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  4. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
  5. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  6. 2025ComPO: Preference Alignment via Comparison Oracles
  7. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs
  8. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment