mlsanityData to 14 Oct 2026

PluralLLM: Pluralistic Alignment in LLMs via Federated Learning

arXiv 2503.09925 ↗First posted 13 Mar 2025cs.LG cs.CLcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Ensuring Large Language Models (LLMs) align with diverse human preferences while preserving privacy and fairness remains a challenge. Existing methods, such as Reinforcement Learning from Human Feedback (RLHF), rely on centralized data collection, making them computationally expensive and privacy-invasive. We introduce PluralLLM a federated learning-based approach that enables multiple user groups to collaboratively train a transformer-based preference predictor without sharing sensitive data, which can also serve as a reward model for aligning LLMs. Our method leverages Federated Averaging (FedAvg) to aggregate preference updates efficiently, achieving 46% faster convergence, a 4% improvement in alignment scores, and nearly the same group fairness measure as in centralized training. Evaluated on a Q/A preference alignment task, PluralLLM demonstrates that federated preference learning offers a scalable and privacy-preserving alternative for aligning LLMs with diverse human values.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 17 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

14 of 17 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024AGR: Age Group fairness Reward for Bias Mitigation in LLMs6 Sep
  2. 2024A Roadmap to Pluralistic Alignment7 Feb

Other references · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  6. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  7. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  8. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  9. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  10. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  11. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  12. 2025FinP: Fairness-in-Privacy in Federated Learning by Addressing Disparities in Privacy Riskcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2026Positive Alignment: Artificial Intelligence for Human Flourishingcited by 5
  2. 2025A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMscited by 3
  3. 2025Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemblecited by 2
  4. 2025FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Modelscited by 1
  5. 2026Overton Pluralistic Reinforcement Learning for Large Language Modelscited by 1
  6. 2026APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMscited by 1
  7. 2026Jailbreaking Large Language Models with Morality Attackscited by 1
  8. 2026Constructive Alignment: Governing Preference Dynamics in Human-AI Interactioncited by 1
  9. 2025Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policiescited by 0
  10. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Modelscited by 0
  11. 2026FedDetox: Robust Federated SLM Alignment via On-Device Data Sanitizationcited by 0
  12. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Modelscited by 0
2 more
  1. 2026Demographic Pluralism: Inference-Time Modeling of Pluralistic Human Preference Distributionscited by 0
  2. 2026Federated Agent Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
  2. 2024Towards Federated RLHF with Aggregated Client Preference for LLMs
  3. 2024Aligning LLMs with Individual Preferences via Interaction
  4. 2026Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
  5. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  6. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  7. 2024Aligning to Thousands of Preferences via System Message Generalization
  8. 2026Personalized Federated Direct Preference Optimization via LoRA Adapters