# PluralLLM: Pluralistic Alignment in LLMs via Federated Learning

arXiv: https://arxiv.org/abs/2503.09925 · HTML: https://mlsanity.com/paper/2503.09925/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning](https://mlsanity.com/paper/2309.00363/) (2023)
- [MaxMin-RLHF: Alignment with Diverse Human Preferences](https://mlsanity.com/paper/2402.08925/) (2024)
- [FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model](https://mlsanity.com/paper/2406.17706/) (2024)
- [AGR: Age Group fairness Reward for Bias Mitigation in LLMs](https://mlsanity.com/paper/2409.04340/) (2024)
- [FinP: Fairness-in-Privacy in Federated Learning by Addressing Disparities in Privacy Risk](https://mlsanity.com/paper/2502.17748/) (2025)

## Built on by (most cited first, 14 of 14)

- [Positive Alignment: Artificial Intelligence for Human Flourishing](https://mlsanity.com/paper/2605.10310/) (2026)
- [A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs](https://mlsanity.com/paper/2512.08786/) (2025)
- [Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble](https://mlsanity.com/paper/2509.11311/) (2025)
- [FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Models](https://mlsanity.com/paper/2511.16992/) (2025)
- [Overton Pluralistic Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2602.20759/) (2026)
- [APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs](https://mlsanity.com/paper/2604.04261/) (2026)
- [Jailbreaking Large Language Models with Morality Attacks](https://mlsanity.com/paper/2604.17053/) (2026)
- [Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction](https://mlsanity.com/paper/2607.00001/) (2026)
- [Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policies](https://mlsanity.com/paper/2511.05018/) (2025)
- [Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models](https://mlsanity.com/paper/2602.00485/) (2026)
- [FedDetox: Robust Federated SLM Alignment via On-Device Data Sanitization](https://mlsanity.com/paper/2604.06833/) (2026)
- [Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models](https://mlsanity.com/paper/2605.03426/) (2026)
- [Demographic Pluralism: Inference-Time Modeling of Pluralistic Human Preference Distributions](https://mlsanity.com/paper/2609.38555/) (2026)
- [Federated Agent Optimization](https://mlsanity.com/paper/2610.01195/) (2026)

## Abstract (arXiv)

Ensuring Large Language Models (LLMs) align with diverse human preferences while preserving privacy and fairness remains a challenge. Existing methods, such as Reinforcement Learning from Human Feedback (RLHF), rely on centralized data collection, making them computationally expensive and privacy-invasive. We introduce PluralLLM a federated learning-based approach that enables multiple user groups to collaboratively train a transformer-based preference predictor without sharing sensitive data, which can also serve as a reward model for aligning LLMs. Our method leverages Federated Averaging (FedAvg) to aggregate preference updates efficiently, achieving 46% faster convergence, a 4% improvement in alignment scores, and nearly the same group fairness measure as in centralized training. Evaluated on a Q/A preference alignment task, PluralLLM demonstrates that federated preference learning offers a scalable and privacy-preserving alternative for aligning LLMs with diverse human values.