mlsanityData to 7 Oct 2026

Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging

arXiv 2310.11564 ↗First posted 17 Oct 2023cs.CLcited by 271

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

While Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with general, aggregate human preferences, it is suboptimal for learning diverse, individual perspectives. In this work, we study Reinforcement Learning from Personalized Human Feedback (RLPHF) problem, wherein LLMs are aligned to multiple (sometimes conflicting) preferences by modeling alignment as a Multi-Objective Reinforcement Learning (MORL) problem. Compared to strong single-objective baselines, we show that we can achieve personalized alignment by decomposing preferences into multiple dimensions. These dimensions are defined based on personalizations that are declared as desirable by the user. In this work, we show that they can be efficiently trained independently in a distributed manner and combined effectively post-hoc through parameter merging. The code is available at https://github.com/joeljang/RLPHF.

Key facts. Cited by 271 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 63 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

44 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  2. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar
  3. 2020Learning to summarize from human feedback2 Sep

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2022Editing Models with Task Arithmeticcited by 1,076
  11. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  12. 2023Whose Opinions Do Language Models Reflect?cited by 759
29 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Instruction Tuning with GPT-4cited by 676
  3. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  6. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  7. 2022Teaching language models to support answers with verified quotescited by 290
  8. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  9. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  10. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  11. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  12. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  13. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  14. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  15. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  16. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
  17. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  18. 2018Training Millions of Personalized Dialogue Agentscited by 126
  19. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  20. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  21. 2019Personalized Dialogue Generation with Diversified Traitscited by 73
  22. 2020Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewardscited by 60
  23. 2019Generating Personalized Recipes from Historical User Preferencescited by 52
  24. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  25. 2019Towards Controllable and Personalized Review Generationcited by 24
  26. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  27. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  28. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  29. 2022Dynamic Planning in Open-Ended Dialogue using Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 90; 2025: 107; 2026: 69.

271 papers.

Sort:

By area:

  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  4. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  5. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  6. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  8. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  9. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  10. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  11. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  12. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
38 more
  1. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  2. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  3. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  4. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  5. 2024Can LLM be a Personalized Judge?cited by 99
  6. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  7. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  8. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  9. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  10. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  11. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  12. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  13. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  14. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  15. 2024Decoding-time Realignment of Language Modelscited by 67
  16. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  17. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 59
  18. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  19. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  20. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 56
  21. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  22. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  23. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  24. 2024Personality Alignment of Large Language Modelscited by 36
  25. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  26. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  27. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  28. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  29. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32
  30. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
  31. 2025Personalized Text Generation with Contrastive Activation Steeringcited by 30
  32. 2025LaMP-QA: A Benchmark for Personalized Long-form Question Answeringcited by 30
  33. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  34. 2025Language Model Personalization via Reward Factorizationcited by 27
  35. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  36. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  37. 2025Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generationcited by 26
  38. 2025ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generationcited by 26

50 most cited of 271 All 271 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  2. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  5. 2024Investigating on RLHF methodology
  6. 2025Doubly Robust Alignment for Large Language Models
  7. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  8. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models