# Distributed Neural Policy Gradient Algorithm for Global Convergence of Networked Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.24113 · HTML: https://mlsanity.com/paper/2505.24113/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2025.

## Built on by (most cited first, 5 of 5)

- [A Communication-Efficient Decentralized Actor-Critic Algorithm](https://mlsanity.com/paper/2510.19199/) (2025)
- [Research on Optimal Control Problem Based on Reinforcement Learning under Knightian Uncertainty](https://mlsanity.com/paper/2506.13207/) (2025)
- [HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization](https://mlsanity.com/paper/2603.03741/) (2026)
- [Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2605.15697/) (2026)
- [Fully Byzantine-Resilient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2609.25701/) (2026)

## Abstract (arXiv)

This paper studies the networked multi-agent reinforcement learning (NMARL) problem, where the objective of agents is to collaboratively maximize the discounted average cumulative rewards. Different from the existing methods that suffer from poor expression due to linear function approximation, we propose a distributed neural policy gradient algorithm that features two innovatively designed neural networks, specifically for the approximate Q-functions and policy functions of agents. This distributed neural policy gradient algorithm consists of two key components: the distributed critic step and the decentralized actor step. In the distributed critic step, agents receive the approximate Q-function parameters from their neighboring agents via a time-varying communication networks to collaboratively evaluate the joint policy. In contrast, in the decentralized actor step, each agent updates its local policy parameter solely based on its own approximate Q-function. In the convergence analysis, we first establish the global convergence of agents for the joint policy evaluation in the distributed critic step. Subsequently, we rigorously demonstrate the global convergence of the overall distributed neural policy gradient algorithm with respect to the objective function. Finally, the effectiveness of the proposed algorithm is demonstrated by comparing it with a centralized algorithm through simulation in the robot path planning environment.