mlsanityData to 7 Oct 2026

Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences

arXiv 2606.07629 ↗First posted 30 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Current approaches to aligning large language models (LLMs) aggregate diverse human preferences into a single reward signal, effectively optimizing for a hypothetical ``average user'' who represents no real person particularly well. This position paper argues that LLMs should learn personalized, individual preferences rather than aggregated ones. We show that aggregation masks critical information about preference diversity, individual values, and contextual dependencies, which is a limitation both theoretically grounded in social choice theory and empirically evident across demographic groups. We analyze the rich structure that human preferences encode, survey technical approaches to personalization, and systematically address counterarguments on scalability, shared standards, and manipulation risk. While personalization introduces genuine safety challenges including filter bubbles, value lock-in, and psychological manipulation, we argue these are manageable through bounded personalization frameworks that preserve universal safety constraints while accommodating legitimate individual variation. We conclude with a concrete research and policy agenda for developing preference-aware models that respect both individual autonomy and collective safety.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 76 references that are in the map. First posted on arXiv 30 May 2026.

Builds on

54 of 76 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Personalized Benchmarking: Evaluating LLMs by Individual Preferences21 Apr
  2. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling12 Feb
  3. 2025Aligning LLMs by Predicting Preferences from User Writing Samples27 May
  4. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation29 Apr
  5. 2024Personalized Adaptation via In-Context Preference Learning17 Oct
  6. 2024Aligning LLM Agents by Learning Latent Preference from User Edits23 Apr
  7. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback16 Apr
  8. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  11. 2023Whose Opinions Do Language Models Reflect?cited by 759
  12. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
34 more
  1. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  3. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  4. 2021Hash Layers For Large Sparse Modelscited by 236
  5. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
  6. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  7. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 145
  8. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 137
  9. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
  10. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  11. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  12. 2023Eliciting Human Preferences with Language Modelscited by 90
  13. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  14. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  15. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  16. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  17. 2017A Voting-Based System for Ethical Decision Makingcited by 55
  18. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  19. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  20. 2024User Embedding Model for Personalized Language Promptingcited by 28
  21. 2024Context Steering: Controllable Personalization at Inference Timecited by 22
  22. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignmentcited by 22
  23. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
  24. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 20
  25. 2026How LLMs Distort Our Written Languagecited by 20
  26. 2026Learning Personalized Agents from Human Feedbackcited by 16
  27. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  28. 2024Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigationcited by 7
  29. 2024RATE: Causal Explainability of Reward Models with Imperfect Counterfactualscited by 6
  30. 2025Meta-Learning for Cold-Start Personalization in Prompt-Tuned LLMscited by 5
  31. 2026SPRInG: Continual LLM Personalization via Selective Parametric Adaptation and Retrieval-Interpolated Generationcited by 4
  32. 2026Beyond Fixed Psychological Personas: State Beats Trait, but Language Models are State-Blindcited by 4
  33. 2025Quantifying the Potential to Escape Filter Bubbles: A Behavior-Aware Measure via Contrastive Simulationcited by 2
  34. 2025From Delegates to Trustees: How Optimizing for Long-Term Interests Shapes Bias and Alignment in LLMcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning
  2. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  3. 2024PersonalLLM: Tailoring LLMs to Individual Preferences
  4. 2026Text as a Universal Interface for Transferable Personalization
  5. 2024Unsupervised Human Preference Learning
  6. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Models
  7. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Models
  8. 2026Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs