# Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus

arXiv: https://arxiv.org/abs/1903.09255 · HTML: https://mlsanity.com/paper/1903.09255/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2019.

## Built on by (most cited first, 11 of 11)

- [Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis](https://mlsanity.com/paper/2109.03699/) (2021)
- [Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games](https://mlsanity.com/paper/2202.09422/) (2022)
- [Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11692/) (2021)
- [Decentralized Deterministic Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.09745/) (2021)
- [One Policy is Enough: Parallel Exploration with a Single Policy is Near-Optimal for Reward-Free Reinforcement Learning](https://mlsanity.com/paper/2205.15891/) (2022)
- [Finite-Time Error Bounds for Distributed Linear Stochastic Approximation](https://mlsanity.com/paper/2111.12665/) (2021)
- [Federated Reinforcement Learning at the Edge](https://mlsanity.com/paper/2112.05908/) (2021)
- [A finite time analysis of distributed Q-learning](https://mlsanity.com/paper/2405.14078/) (2024)
- [Cooperative Actor-Critic via TD Error Aggregation](https://mlsanity.com/paper/2207.12533/) (2022)
- [Achieving Collective Welfare in Multi-Agent Reinforcement Learning via Suggestion Sharing](https://mlsanity.com/paper/2412.12326/) (2024)
- [Variance-Reduced Q-Learning over Static and Time-Varying Networks](https://mlsanity.com/paper/2607.21876/) (2026)

## Abstract (arXiv)

In this paper, we propose a distributed off-policy actor critic method to solve multi-agent reinforcement learning problems. Specifically, we assume that all agents keep local estimates of the global optimal policy parameter and update their local value function estimates independently. Then, we introduce an additional consensus step to let all the agents asymptotically achieve agreement on the global optimal policy function. The convergence analysis of the proposed algorithm is provided and the effectiveness of the proposed algorithm is validated using a distributed resource allocation example. Compared to relevant distributed actor critic methods, here the agents do not share information about their local tasks, but instead they coordinate to estimate the global policy function.