mlsanityData to 7 Oct 2026

CoPL: Collaborative Preference Learning for Personalizing LLMs

arXiv 2503.01658 ↗First posted 3 Mar 2025cs.LG cs.AI cs.IRcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Personalizing large language models (LLMs) is important for aligning outputs with diverse user preferences, yet existing methods struggle with flexibility and generalization. We propose CoPL (Collaborative Preference Learning), a graph-based collaborative filtering framework that models user-response relationships to enhance preference estimation, particularly in sparse annotation settings. By integrating a mixture of LoRA experts, CoPL efficiently fine-tunes LLMs while dynamically balancing shared and user-specific preferences. Additionally, an optimization-free adaptation strategy enables generalization to unseen users without fine-tuning. Experiments on UltraFeedback-P demonstrate that CoPL outperforms existing personalized reward models, effectively capturing both common and controversial preferences, making it a scalable solution for personalized LLM alignment. The code is available at https://github.com/ml-postech/CoPL.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 46 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

31 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?18 Oct
  2. 2024A Roadmap to Pluralistic Alignment7 Feb
  3. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,134
  6. 2019Neural Graph Collaborative Filteringcited by 660
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  8. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  9. 2017Neural Factorization Machines for Sparse Predictive Analyticscited by 183
  10. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  11. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  12. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
14 more
  1. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  2. 2020Revisiting Graph based Collaborative Filtering: A Linear Residual Graph Convolutional Network Approachcited by 119
  3. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  4. 2022Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learningcited by 86
  5. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  6. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  7. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  8. 2024CollabStory: Multi-LLM Collaborative Story Generation and Authorship Analysiscited by 26
  9. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoEcited by 25
  10. 2021Fairness-aware Class Imbalanced Learningcited by 19
  11. 2024CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMscited by 17
  12. 2022Coarse-to-Fine Sparse Sequential Recommendationcited by 8
  13. 2024Leveraging LLM Tutoring Systems for Non-Native English Speakers in Introductory CS Coursescited by 8
  14. 2024How Does Message Passing Improve Collaborative Filtering?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modelingcited by 2
  2. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalizationcited by 1
  3. 2026Efficient Personalization of Generative User Interfacescited by 0
  4. 2026Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrastcited by 0
  5. 2026Learning to Adapt Cross-Domain Preferences via Meta-LoRA for LLM Personalizationcited by 0
  6. 2026hoBIT: A Profile-Aware Retrieval-Augmented Chatbot for University Academic Advisingcited by 0
  7. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0
  8. 2026Using Context Is Not Enough: Test-Time Training for Personalized Reward Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PersonalLLM: Tailoring LLMs to Individual Preferences
  2. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modeling
  3. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  4. 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework
  5. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  6. 2024Orchestrating LLMs with Different Personalizations
  7. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
  8. 2024Aligning LLMs with Individual Preferences via Interaction