mlsanityData to 14 Oct 2026

Efficient Federated RLHF via Zeroth-Order Policy Optimization

arXiv 2604.17747 ↗First posted 20 Apr 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This paper considers reinforcement learning from human feedback in a federated learning setting with resource-constrained agents, such as edge devices. We propose an efficient federated RLHF algorithm, named Partitioned, Sign-based Stochastic Zeroth-order Policy Optimization (Par-S2ZPO). The algorithm is built on zeroth-order optimization with binary perturbation, resulting in low communication, computation, and memory complexity by design. Our theoretical analysis establishes an upper bound on the convergence rate of Par-S2ZPO, revealing that it is as efficient as its centralized counterpart in terms of sample complexity but converges faster in terms of policy update iterations. Our experimental results show that it outperforms a FedAvg-based RLHF on four MuJoCo RL tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 22 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

15 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback17 Jul
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  5. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  6. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  7. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  8. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  9. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 102
  10. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  11. 2024FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHFcited by 12
  12. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learning
  2. 2025On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environments
  3. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning
  4. 2026Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Cost
  5. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis
  6. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling
  7. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learning
  8. 2024Federated reinforcement learning for robot motion planning with zero-shot generalization