mlsanityData to 7 Oct 2026

Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedback

arXiv 2605.15697 ↗First posted 15 May 2026cs.MAcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We study a networked multi-agent reinforcement learning (NMARL) problem with human feedback in an infinite-horizon setting, where agents interact over an underlying network with localized state dependencies and aim to collaboratively maximize the average discounted return. Existing approaches with preference feedback are primarily developed for single-agent settings and rely on centralized training, which limits their scalability and applicability to large-scale networked multi-agent systems. To address this, we introduce a novel human feedback mechanism based on spatiotemporally truncated trajectories, defined as H-horizon trajectory pairs aggregated over each agent's kappa-hop neighborhood. Building on this, we develop a distributed zeroth-order policy gradient algorithm, where each agent estimates its local policy gradient using human preference feedback generated from both the current joint policy and a perturbed joint policy drawn from zero-mean Gaussian distribution. Specifically, the algorithm is fully distributed, as the feedback received by each agent depends solely on the state-action information within its kappa-hop neighborhood and does not require explicit reward signals or centralized control. We further rigorously establish that the proposed algorithm converges to an ε-stationary point with polynomial sample complexity. Finally, simulation results in a stochastic GridWorld environment and a predator-prey environment further demonstrate that the effectiveness and scalability of the proposed algorithm in achieving collaborative optimization based solely on human preference feedback.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 39 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

21 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback24 Feb
  2. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference25 Sep
  3. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization15 Feb
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  3. 2021Reinforcement learning based recommender systems: A surveycited by 187
  4. 2019Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Controlcited by 159
  5. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  6. 2020Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Rewardcited by 46
  7. 2019Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learningcited by 33
  8. 2020Multi-Agent Reinforcement Learning in Stochastic Networked Systemscited by 30
  9. 2022On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Gamescited by 22
  10. 2021A Survey on Reinforcement Learning for Recommender Systemscited by 17
  11. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  12. 2023Convergence Rates for Localized Actor-Critic in Networked Markov Potential Gamescited by 9
4 more
  1. 2024How language models extrapolate outside the training data: A case study in Textualized Gridworldcited by 7
  2. 2020Fully Asynchronous Policy Evaluation in Distributed Reinforcement Learning over Networkscited by 6
  3. 2025Distributed Neural Policy Gradient Algorithm for Global Convergence of Networked Multi-Agent Reinforcement Learningcited by 4
  4. 2024Reactive Multi-Robot Navigation in Outdoor Environments Through Uncertainty-Aware Active Learning of Human Preference Landscapecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
  2. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning
  3. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning
  4. 2021Offline Decentralized Multi-Agent Reinforcement Learning
  5. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition
  6. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  7. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback
  8. 2025Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarm