mlsanityData to 14 Oct 2026

FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF

arXiv 2412.15538 ↗First posted 20 Dec 2024cs.LG cs.AI cs.CRcited by 12

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

In the era of increasing privacy concerns and demand for personalized experiences, traditional Reinforcement Learning with Human Feedback (RLHF) frameworks face significant challenges due to their reliance on centralized data. We introduce Federated Reinforcement Learning with Human Feedback (FedRLHF), a novel framework that decentralizes the RLHF process. FedRLHF enables collaborative policy learning across multiple clients without necessitating the sharing of raw data or human feedback, thereby ensuring robust privacy preservation. Leveraging federated reinforcement learning, each client integrates human feedback locally into their reward functions and updates their policies through personalized RLHF processes. We establish rigorous theoretical foundations for FedRLHF, providing convergence guarantees, and deriving sample complexity bounds that scale efficiently with the number of clients. Empirical evaluations on the MovieLens and IMDb datasets demonstrate that FedRLHF not only preserves user privacy but also achieves performance on par with centralized RLHF, while enhancing personalization across diverse client environments.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 48 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

23 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  2. 2020Learning to summarize from human feedback2 Sep

Other references · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  9. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  10. 2017Federated Multi-Task Learningcited by 672
  11. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  12. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
9 more
  1. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  2. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  3. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 78
  4. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  5. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  6. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  7. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 36
  8. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  9. 2024CAESAR: Enhancing Federated RL in Heterogeneous MDPs through Convergence-Aware Sampling with Screeningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  2. 2026FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewardscited by 2
  3. 2026Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocolcited by 2
  4. 2026APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMscited by 1
  5. 2026Finding Stationary Points by Comparisonscited by 1
  6. 2026FedPref: Federated Preference Learning for Structured Radiology Report Extractioncited by 1
  7. 2026Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimizationcited by 1
  8. 2025Position Paper: Rethinking Privacy in RL for Sequential Decision-making in the Age of LLMscited by 0
  9. 2026Efficient Federated RLHF via Zeroth-Order Policy Optimizationcited by 0
  10. 2026Global Federated Learning Strategies for Building Efficient Personalized Modelscited by 0
  11. 2026Authority-Preserving Evaluation of Medical Vision-Language Assistantscited by 0
  12. 2026Federated Agent Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards User-level Private Reinforcement Learning with Human Feedback
  2. 2024Towards Federated RLHF with Aggregated Client Preference for LLMs
  3. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
  4. 2025Approximated Behavioral Metric-based State Projection for Federated Reinforcement Learning
  5. 2026Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation
  6. 2025FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation
  7. 2026Privacy Preserving Reinforcement Learning with One-Sided Feedback
  8. 2024Preserving Expert-Level Privacy in Offline Reinforcement Learning