mlsanityData to 7 Oct 2026

Towards Federated RLHF with Aggregated Client Preference for LLMs

arXiv 2407.03038 ↗First posted 3 Jul 2024cs.CL cs.DC cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Reinforcement learning with human feedback (RLHF) fine-tunes a pretrained large language model (LLM) using user preference data, enabling it to generate content aligned with human preferences. However, due to privacy concerns, users may be reluctant to share sensitive preference data. To address this, we propose utilizing Federated Learning (FL) techniques, allowing large-scale preference collection from diverse real-world users without requiring them to transmit data to a central server. Our federated RLHF methods (i.e., FedBis and FedBiscuit) encode each client's preferences into binary selectors and aggregate them to capture common preferences. In particular, FedBiscuit overcomes key challenges, such as preference heterogeneity and reward hacking, through innovative solutions like grouping clients with similar preferences to reduce heterogeneity and using multiple binary selectors to enhance LLM output quality. To evaluate the performance of the proposed methods, we establish the first federated RLHF benchmark with a heterogeneous human preference dataset. Experimental results show that by integrating the LLM with aggregated client preferences, FedBis and FedBiscuit significantly enhance the professionalism and readability of the generated content.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 90 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

62 of 90 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  2. 2020Learning to summarize from human feedback2 Sep

Other references · 60

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
38 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  3. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  12. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  13. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  14. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  15. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  16. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  17. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  18. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  19. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  20. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  21. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  22. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  23. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  24. 2023Generative Judge for Evaluating Alignmentcited by 163
  25. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  26. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  27. 2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 127
  28. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  29. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
  30. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  31. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  32. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  33. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  34. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 68
  35. 2021LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learningcited by 63
  36. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  37. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  38. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 4.

11 papers.

  1. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  2. 2025Talk to Right Specialists: Iterative Routing in Multi-agent Systems for Question Answeringcited by 6
  3. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  4. 2025PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoningcited by 4
  5. 2025Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agentscited by 3
  6. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  7. 2025Towards Privacy-Preserving and Heterogeneity-aware Split Federated Learning via Probabilistic Maskingcited by 0
  8. 2025FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Modelscited by 0
  9. 2026Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferencescited by 0
  10. 2026QSplitFL: Capability Aware Deep Q-Learning for Optimal Split Point Selection in Split Federated Learningcited by 0
  11. 2026Federated Agent Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
  2. 2025A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs
  3. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
  4. 2026APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs
  5. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  6. 2026Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
  7. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  8. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences