mlsanityData to 14 Oct 2026

Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization

arXiv 2604.00997 ↗First posted 1 Apr 2026cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Reward factorization personalizes large language models (LLMs) by decomposing rewards into shared basis functions and user-specific weights. Yet, existing methods estimate user weights from scarce data in isolation and as deterministic points, leading to inaccurate and unreliable inference. We introduce Variational Reward Factorization (VRF), an uncertainty-aware framework that represents each user's preferences as a variational distribution in a shared preference space. VRF infers user distributions via a variational encoder, derives weights through Wasserstein distance matching with shared probabilistic bases, and downweights uncertain estimates through a variance-attenuated loss. On three benchmarks, VRF outperforms all baselines across seen and unseen users, few-shot scenarios, and varying uncertainty levels, with gains extending to downstream alignment. Our code is available at https://github.com/Gyu-Seok-Lee/VRF_COLM26.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 49 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

35 of 49 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment26 Jan
  2. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs5 Jun
  3. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modeling20 Apr
  4. 2025Language Model Personalization via Reward Factorization8 Mar
  5. 2025CoPL: Collaborative Preference Learning for Personalizing LLMs3 Mar
  6. 2024PersonalLLM: Tailoring LLMs to Individual Preferences30 Sep
  7. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization28 May
  8. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  9. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
  10. 2020Learning to summarize from human feedback2 Sep

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  12. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
13 more
  1. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  2. 2013Expectation Propagation for approximate Bayesian inferencecited by 174
  3. 2023Uncovering ChatGPT's Capabilities in Recommender Systemscited by 161
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  5. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  6. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 74
  7. 2023Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Modelscited by 45
  8. 2025Personalized Text Generation with Contrastive Activation Steeringcited by 34
  9. 2024LLMs + Persona-Plug = Personalized LLMscited by 32
  10. 2024User Embedding Model for Personalized Language Promptingcited by 28
  11. 2023BookGPT: A General Framework for Book Recommendation Empowered by Large Language Modelcited by 22
  12. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 11
  13. 2026Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedbackcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
  2. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  3. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalization
  4. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences
  5. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
  6. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  7. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users
  8. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling