mlsanityData to 7 Oct 2026

LoRe: Personalizing LLMs via Low-Rank Reward Modeling

arXiv 2504.14439 ↗First posted 20 Apr 2025cs.LG cs.AI cs.CLcited by 34

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Personalizing large language models (LLMs) to accommodate diverse user preferences is essential for enhancing alignment and user satisfaction. Traditional reinforcement learning from human feedback (RLHF) approaches often rely on monolithic value representations, limiting their ability to adapt to individual preferences. We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and generalize user-specific reward functions. By representing reward functions in a low-dimensional subspace and modeling individual preferences as weighted combinations of shared basis functions, our approach avoids rigid user categorization while enabling scalability and few-shot adaptation. We validate our method on multiple preference datasets, demonstrating superior generalization to unseen users and improved accuracy in preference prediction tasks.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 48 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

43 of 48 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users26 Feb
  2. 2024Test-Time Alignment via Hypothesis Reweighting11 Dec
  3. 2024PAD: Personalized Alignment of LLMs at Decoding-Time5 Oct
  4. 2024PersonalLLM: Tailoring LLMs to Individual Preferences30 Sep
  5. 2024Personality Alignment of Large Language Models21 Aug
  6. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment24 Jul
  7. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  8. 2024A Roadmap to Pluralistic Alignment7 Feb
  9. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  10. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  11. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  12. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
1 more
  1. 2020Learning to summarize from human feedback2 Sep

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
18 more
  1. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  2. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  3. 2023Personality Traits in Large Language Modelscited by 166
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  5. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 122
  6. 2024Can LLM be a Personalized Judge?cited by 99
  7. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  8. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  9. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  10. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  11. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
  12. 2024Personalized Large Language Modelscited by 44
  13. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 36
  14. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  15. 2024Offline Multi-task Transfer RL with Representational Penalizationcited by 13
  16. 2015Preference Completion: Large-scale Collaborative Ranking from Pairwise Comparisonscited by 11
  17. 2023Initializing Services in Interactive ML Systems for Diverse Userscited by 5
  18. 2014Individualized Rank Aggregation using Nuclear Norm Regularizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 23.

34 papers.

Sort:

By area:

  1. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
  2. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  3. 2025A Personalized Conversational Benchmark: Towards Simulating Personalized Conversationscited by 16
  4. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 15
  5. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  6. 2025PrefDisco: Benchmarking Proactive Personalized Reasoningcited by 7
  7. 2025Preference Learning for AI Alignment: a Causal Perspectivecited by 6
  8. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inferencecited by 5
  9. 2026Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedbackcited by 5
  10. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  11. 2025The Reward Model Selection Crisis in Personalized Alignmentcited by 3
  12. 2026CURP: Codebook-based Continuous User Representation for Personalized Generation with LLMscited by 3
22 more
  1. 2026EXACT: Explicit Attribute-Guided Decoding-Time Personalizationcited by 3
  2. 2026Cold-Start Personalization via Training-Free Priors from Structured World Modelscited by 2
  3. 2026Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracycited by 2
  4. 2026Preference-Aware Rubric Learning for Personalized Evaluationcited by 2
  5. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modelingcited by 2
  6. 2025Personalized Collaborative Learning with Affinity-Based Variance Reductioncited by 1
  7. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalizationcited by 1
  8. 2025Offline Policy Evaluation of Multi-Turn LLM Health Coaching with Real Userscited by 0
  9. 2026RepUCB: Representation Learning-Based UCB for Heterogeneous Multi-Task Linear Banditscited by 0
  10. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalizationcited by 0
  11. 2026Personalized Alignment Revisited: The Necessity and Sufficiency of User Diversitycited by 0
  12. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  13. 2026LLM Consumer Behavior Theory: Foundations of a Novel Research Fieldcited by 0
  14. 2026AI Alignment From Social Choice Perspectivescited by 0
  15. 2026A Method for Learning Value Systems in Generative AIcited by 0
  16. 2026Cautious Context Steering for Language Model Personalizationcited by 0
  17. 2026PALATE: Personalized Aesthetic Learning through Adaptive Taste Evolution for Multi-User Portrait Retouchingcited by 0
  18. 2026Bounded Personas Match Retrieval on Classification but Not Regression for a Frozen Agentcited by 0
  19. 2026Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Modelscited by 0
  20. 2026Social Choice Foundations for Simulation-Augmented Generationcited by 0
  21. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0
  22. 2026Gradient-Aligned Pair Selection for Personalized Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Language Model Personalization via Reward Factorization
  2. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
  3. 2024Orchestrating LLMs with Different Personalizations
  4. 2025CoPL: Collaborative Preference Learning for Personalizing LLMs
  5. 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework
  6. 2024Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation
  7. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHF
  8. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization