mlsanityData to 7 Oct 2026

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

arXiv 2408.10075 ↗First posted 19 Aug 2024cs.LG cs.AI cs.CLcited by 147

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 71 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

53 of 71 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Pareto-Optimal Learning from Preferences with Hidden Context21 Jun
  2. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  3. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  4. 2023Nash Learning from Human Feedback1 Dec
  5. 2023A density estimation perspective on learning from pairwise human preferences23 Nov
  6. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  7. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback20 Jul
  8. 2023Is RLHF More Difficult than Standard RL?25 Jun
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  12. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2016Variational Inference: A Review for Statisticianscited by 1,466
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
28 more
  1. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  2. 2020Artificial Intelligence, Values and Alignmentcited by 443
  3. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  4. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  6. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  7. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  8. 2024A Roadmap to Pluralistic Alignmentcited by 175
  9. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  10. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  11. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
  12. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  13. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  14. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  15. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  16. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  17. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  18. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  19. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  20. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  21. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  22. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  23. 2020Detecting Out-of-distribution Samples via Variational Auto-encoder with Reliable Uncertainty Estimationcited by 24
  24. 2020LESS is More: Rethinking Probabilistic Models of Human Behaviorcited by 17
  25. 2021Uncertain Decisions Facilitate Better Preference Learningcited by 10
  26. 2023TART: A plug-and-play Transformer module for task-agnostic reasoningcited by 10
  27. 2023Initializing Services in Interactive ML Systems for Diverse Userscited by 5
  28. 2022vec2text with Round-Trip Translationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 73; 2026: 64.

147 papers.

Sort:

By area:

  1. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  2. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 64
  3. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  4. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 44
  5. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  6. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  7. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  8. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  9. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  10. 2025Value Profiles for Encoding Human Variationcited by 29
  11. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  12. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
38 more
  1. 2025Language Model Personalization via Reward Factorizationcited by 27
  2. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  3. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Rewardcited by 25
  4. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  5. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
  6. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  7. 2024On the Way to LLM Personalization: Learning to Remember User Conversationscited by 21
  8. 2024No Preference Left Behind: Group Distributional Preference Optimizationcited by 21
  9. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  10. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  11. 2025PrefPalette: Personalized Preference Modeling with Latent Attributescited by 15
  12. 2025CTR-Driven Advertising Image Generation with Multimodal Large Language Modelscited by 14
  13. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Userscited by 14
  14. 2025Cross-environment Cooperation Enables Zero-shot Multi-agent Coordinationcited by 14
  15. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  16. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  17. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  18. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  19. 2025Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Modelscited by 11
  20. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  21. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  22. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  23. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  24. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 9
  25. 2025Learning to summarize user information for personalized reinforcement learning from human feedbackcited by 9
  26. 2025CoPL: Collaborative Preference Learning for Personalizing LLMscited by 8
  27. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  28. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  29. 2025Clone-Robust AI Alignmentcited by 7
  30. 2025Adaptive Elicitation of Latent Information Using Natural Languagecited by 7
  31. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codescited by 7
  32. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Datacited by 7
  33. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  34. 2025PrefDisco: Benchmarking Proactive Personalized Reasoningcited by 7
  35. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  36. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
  37. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  38. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 6

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2025Influencing Humans to Conform to Preference Models for RLHF
  3. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  4. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  5. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2023A Survey of Reinforcement Learning from Human Feedback
  8. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences