mlsanityData to 7 Oct 2026

Personalized Language Modeling from Personalized Human Feedback

arXiv 2402.05133 ↗First posted 6 Feb 2024cs.CL cs.AI cs.LGcited by 124

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Personalized large language models (LLMs) are designed to tailor responses to individual user preferences. While Reinforcement Learning from Human Feedback (RLHF) is a commonly used framework for aligning LLMs with human preferences, vanilla RLHF assumes that all human preferences share the same distribution, preventing fine-tuned LLMs from generating personalized content when user preferences are diverse. In this work, we propose Personalized-RLHF (P-RLHF), an efficient framework that utilizes a lightweight user model to capture individual user preferences and jointly learns the user model and the personalized LLM from human feedback. P-RLHF exhibits the following three characteristics: (1) It enables an LLM to generate personalized content and scale efficiently with growing number of users. (2) It handles both explicit user preferences described as textual input and implicit user preferences encoded in the feedback data. (3) It eliminates the need for users to fully articulate their preferences, which are normally needed for prompting LLMs to generate personalized content yet are often impractical to obtain in real-world scenarios. Our experimental results show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences, outperforming vanilla, non-personalized RLHF and prompting-based personalization approaches across different tasks. We opensource our code at https://github.com/HumainLab/Personalized_RLHF.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 48 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

36 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  2. 2023Aligning Language Models to User Opinions24 May
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  4. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
19 more
  1. 2023Whose Opinions Do Language Models Reflect?cited by 759
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023Instruction Tuning with GPT-4cited by 676
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  8. 2022Teaching language models to support answers with verified quotescited by 290
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  10. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  11. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  12. 2017Deep learning from crowdscited by 105
  13. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  14. 2023Questioning the Survey Responses of Large Language Modelscited by 70
  15. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  16. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  17. 2023Moral Machine or Tyranny of the Majority?cited by 22
  18. 2023A Survey on Large Language Models for Personalized and Explainable Recommendationscited by 11
  19. 2018Analysis of Minimax Error Rate for Crowdsourcing and Its Application to Worker Clustering Modelcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 58; 2026: 39.

124 papers.

Sort:

By area:

  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  2. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  4. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  5. 2024Can LLM be a Personalized Judge?cited by 99
  6. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  7. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  8. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  9. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  10. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  11. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  12. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
38 more
  1. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 56
  2. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  3. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  4. 2024Personality Alignment of Large Language Modelscited by 36
  5. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  6. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
  7. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  8. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  9. 2025PersonaBench: Evaluating AI Models on Understanding Personal Information through Accessing (Synthetic) Private User Datacited by 26
  10. 2025A Survey of Personalization: From RAG to Agentcited by 26
  11. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  12. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  13. 2025Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generationcited by 23
  14. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
  15. 2024On the Way to LLM Personalization: Learning to Remember User Conversationscited by 21
  16. 2024No Preference Left Behind: Group Distributional Preference Optimizationcited by 21
  17. 2024ComPO: Community Preferences for Language Model Personalizationcited by 19
  18. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  19. 2025PRIME: Large Language Model Personalization with Cognitive Dual-Memory and Personalized Thought Processcited by 16
  20. 2026Learning Personalized Agents from Human Feedbackcited by 16
  21. 2025Personalize Your LLM: Fake it then Align itcited by 15
  22. 2025PrefPalette: Personalized Preference Modeling with Latent Attributescited by 15
  23. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 15
  24. 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 14
  25. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  26. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  27. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  28. 2024FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHFcited by 11
  29. 2025TOM-SWE: User Mental Modeling For Software Engineering Agentscited by 11
  30. 2025Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Modelscited by 10
  31. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  32. 2025ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulationcited by 10
  33. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  34. 2025Learning to summarize user information for personalized reinforcement learning from human feedbackcited by 9
  35. 2025CoPL: Collaborative Preference Learning for Personalizing LLMscited by 8
  36. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  37. 2025Personalized Language Models via Privacy-Preserving Evolutionary Model Mergingcited by 8
  38. 2026Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactionscited by 8

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Language Model Personalization via Reward Factorization
  2. 2024Orchestrating LLMs with Different Personalizations
  3. 2024Unsupervised Human Preference Learning
  4. 2024Personalized Adaptation via In-Context Preference Learning
  5. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferences
  6. 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework
  7. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
  8. 2024Investigating on RLHF methodology