mlsanityData to 7 Oct 2026

Language Model Personalization via Reward Factorization

arXiv 2503.06358 ↗First posted 8 Mar 2025cs.LGcited by 27

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Modern large language models (LLMs) are optimized for human-aligned responses using Reinforcement Learning from Human Feedback (RLHF). However, existing RLHF approaches assume a universal preference model and fail to account for individual user preferences, limiting their effectiveness in personalized applications. We introduce a framework that extends RLHF to enable user personalization by leveraging the assumption that user preferences lie in a low-dimensional space. Instead of training a separate model per user, we represent user-specific rewards as a linear combination of base reward functions. Using only ~10 user responses, our method can infer user-specific rewards and align LLM outputs accordingly. We validate our approach through experiments with both synthetic and real users, demonstrating significant personalization achieved by our method. In human evaluations, our method achieves a 67% win rate over default GPT-4o responses.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 47 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

36 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024PAD: Personalized Alignment of LLMs at Decoding-Time5 Oct
  2. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment24 Jul
  3. 2024A Roadmap to Pluralistic Alignment7 Feb
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  5. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  6. 2020Learning to summarize from human feedback2 Sep

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2020A Survey of Deep Active Learningcited by 547
  10. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  12. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
18 more
  1. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  4. 2023Controlled Decoding from Language Modelscited by 133
  5. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
  6. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  7. 2024Personalization of Large Language Models: A Surveycited by 100
  8. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  9. 2024Can LLM be a Personalized Judge?cited by 99
  10. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  11. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  12. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  13. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  14. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  15. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  16. 2023Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Modelscited by 40
  17. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
  18. 2024Understanding the Role of User Profile in the Personalization of Large Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 21.

27 papers.

Sort:

By area:

  1. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Rewardcited by 25
  2. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  3. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  4. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 6
  5. 2026Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedbackcited by 5
  6. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scalingcited by 4
  7. 2025The Reward Model Selection Crisis in Personalized Alignmentcited by 3
  8. 2026CARD: Cluster-level Adaptation with Reward-guided Decoding for Personalized Text Generationcited by 2
  9. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
  10. 2026Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracycited by 2
  11. 2026Preference-Aware Rubric Learning for Personalized Evaluationcited by 2
  12. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalizationcited by 1
15 more
  1. 2025Prompt Mechanisms in Medical Imaging: A Comprehensive Surveycited by 0
  2. 2026Personalizing black-box models for nonparametric regression with minimax optimalitycited by 0
  3. 2026Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMscited by 0
  4. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalizationcited by 0
  5. 2026Alignment has a Fantasia Problemcited by 0
  6. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
  7. 2026Personalized Alignment Revisited: The Necessity and Sufficiency of User Diversitycited by 0
  8. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  9. 2026Inverse RL Helps Align AI by Imitating Humanscited by 0
  10. 2026Cautious Context Steering for Language Model Personalizationcited by 0
  11. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimizationcited by 0
  12. 2026Using Context Is Not Enough: Test-Time Training for Personalized Reward Modelingcited by 0
  13. 2026Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Modelscited by 0
  14. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0
  15. 2026Gradient-Aligned Pair Selection for Personalized Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modeling
  2. 2024Personalized Language Modeling from Personalized Human Feedback
  3. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
  4. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  5. 2025Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learning
  6. 2024Orchestrating LLMs with Different Personalizations
  7. 2024Personalized Adaptation via In-Context Preference Learning
  8. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning