mlsanityData to 7 Oct 2026

A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications

arXiv 2503.17003 ↗First posted 21 Mar 2025cs.CLcited by 40

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their transition to real-world applications reveals a critical limitation: the inability to adapt to individual preferences while maintaining alignment with universal human values. Current alignment techniques adopt a one-size-fits-all approach that fails to accommodate users' diverse backgrounds and needs. This paper presents the first comprehensive survey of personalized alignment-a paradigm that enables LLMs to adapt their behavior within ethical boundaries based on individual preferences. We propose a unified framework comprising preference memory management, personalized generation, and feedback-based alignment, systematically analyzing implementation approaches and evaluating their effectiveness across various scenarios. By examining current techniques, potential risks, and future challenges, this survey provides a structured foundation for developing more adaptable and ethically-aligned LLMs.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 121 of its 216 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

121 of 216 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning26 Feb
  2. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferences20 Feb
  3. 2024FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF20 Dec
  4. 2024ComPO: Community Preferences for Language Model Personalization21 Oct
  5. 2024Aligning LLMs with Individual Preferences via Interaction4 Oct
  6. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment24 Jul
  7. 2024Collective Constitutional AI: Aligning a Language Model with Public Input12 Jun
  8. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  9. 2024AI Alignment with Changing and Influenceable Reward Functions28 May
  10. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  12. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
5 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Models12 Oct
  2. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback2 Oct
  4. 2022In conversation with Artificial Intelligence: aligning language models with human values1 Sep
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 104

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2017Neural Collaborative Filteringcited by 1,108
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2023Whose Opinions Do Language Models Reflect?cited by 759
  9. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  12. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  13. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  14. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  15. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  16. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  17. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  18. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  19. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 336
  20. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  21. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  22. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  23. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  24. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  25. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  26. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  27. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  28. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  29. 2023Is ChatGPT a Good Recommender? A Preliminary Studycited by 192
  30. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  31. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  32. 2023Text Is All You Need: Learning Language Representations for Sequential Recommendationcited by 149
  33. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  34. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  35. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  36. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  37. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  38. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 30.

40 papers.

Sort:

By area:

  1. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  2. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  3. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 15
  4. 2025Towards Personalized Deep Research: Benchmarks and Evaluationscited by 14
  5. 2026AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignmentcited by 6
  6. 2025Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signalscited by 5
  7. 2025FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Datacited by 5
  8. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inferencecited by 5
  9. 2026PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harmcited by 5
  10. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  11. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scalingcited by 4
  12. 2026AI Alignment Breaks at the Edgecited by 3
28 more
  1. 2026Text as a Universal Interface for Transferable Personalizationcited by 2
  2. 2026AI and My Values: User Perceptions of LLMs' Ability to Extract, Embody, and Explain Human Values from Casual Conversationscited by 2
  3. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  4. 2025Classical AI vs. LLMs for Decision-Maker Alignment in Health Insurance Choicescited by 1
  5. 2026CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Taskscited by 1
  6. 2026CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inferencecited by 1
  7. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignmentcited by 1
  8. 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Frameworkcited by 1
  9. 2026To Whom Do Language Models Align? Measuring Principal Hierarchies Under High-Stakes Competing Demandscited by 1
  10. 2026Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialoguecited by 1
  11. 2025RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Modelscited by 0
  12. 2026Situation Graph Prediction for User Perspective Modelingcited by 0
  13. 2026EpiPersona: Persona Projection and Episode Coupling for Pluralistic Preference Modelingcited by 0
  14. 2026One Model for All: Multi-Objective Controllable Language Modelscited by 0
  15. 2026When to Ask a Question: Understanding Communication Strategies in Generative AI Toolscited by 0
  16. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Modelscited by 0
  17. 2026From Empathy to Personalized Empathy: Adapting Empathetic Strategies to Individual Userscited by 0
  18. 2026Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Modelscited by 0
  19. 2026Toward Human-Centered Multi-Agent Systems: Integrating Cognition, Culture, Values, and Cooperation in AI Agentscited by 0
  20. 2026LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logscited by 0
  21. 2026Evaluating the Hidden Costs of Personalization in Large Language Modelscited by 0
  22. 2026Language, Language Models, and What We're Talking Aboutcited by 0
  23. 2026Aplaud: Adaptive Personalized Low-Rank Decomposition for User-Specific LLMcited by 0
  24. 2026HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalizationcited by 0
  25. 2026Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generationcited by 0
  26. 2026One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extractioncited by 0
  27. 2026COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluationcited by 0
  28. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  3. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  4. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  5. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  6. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback
  7. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  8. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models