mlsanityData to 7 Oct 2026

Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences

arXiv 2605.30873 ↗First posted 29 May 2026cs.LG cs.AI cs.DCcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Federated Learning (FL) offers a privacy-preserving pathway for aligning Large Language Models (LLMs); however, existing frameworks typically enforce a monolithic reward model, inevitably averaging out inherently conflicting user preferences (e.g., helpfulness vs. harmlessness). While Variational Preference Learning (VPL) offers a pathway to personalization, adapting it to decentralized settings presents a fundamental challenge: posterior collapse driven by severe local data scarcity and heterogeneity. In this paper, we propose Federated Variational Preference Alignment with Gumbel-Softmax Prior (FedVPA-GP), a framework designed to disentangle diverse preferences without compromising privacy. To stabilize variational inference, we introduce a Federated Mixture Prior that enables clients to leverage the aggregate population distribution as a dynamic prior. Furthermore, we incorporate an Orthogonal Loss that explicitly enforces the separation of preference prototypes in the latent space. Experiments on the HH-RLHF dataset demonstrate that FedVPA-GP significantly outperforms monolithic baselines, successfully disentangling conflicting user intents and enabling dynamic preference switching.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 33 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

22 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Towards Federated RLHF with Aggregated Client Preference for LLMs3 Jul
  2. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2023Whose Opinions Do Language Models Reflect?cited by 759
8 more
  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  2. 2015Generating Sentences from a Continuous Spacecited by 607
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  5. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  6. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  7. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  8. 2024Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
  2. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
  3. 2026APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs
  4. 2026Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
  5. 2025A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs
  6. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
  7. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences
  8. 2026PAPA: Online Personalized Active Preference Alignment