# Multi-agent Off-policy Actor-Critic Reinforcement Learning for Partially Observable Environments

arXiv: https://arxiv.org/abs/2407.04974 · HTML: https://mlsanity.com/paper/2407.04974/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 6 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning](https://mlsanity.com/paper/1503.04269/) (2015)
- [On Convergence of Emphatic Temporal-Difference Learning](https://mlsanity.com/paper/1506.02582/) (2015)
- [A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning](https://mlsanity.com/paper/1903.06372/) (2019)
- [Social Learning and Distributed Hypothesis Testing](https://mlsanity.com/paper/1410.4307/) (2014)
- [Online Meta-Critic Learning for Off-Policy Actor-Critic Methods](https://mlsanity.com/paper/2003.05334/) (2020)
- [Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability](https://mlsanity.com/paper/2301.01649/) (2023)
- [SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2105.06228/) (2021)
- [Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention](https://mlsanity.com/paper/2304.12653/) (2023)
- [Cooperative Multi-Agent Reinforcement Learning with Partial Observations](https://mlsanity.com/paper/2006.10822/) (2020)
- [Learning from Heterogeneous Data Based on Social Interactions over Graphs](https://mlsanity.com/paper/2112.09483/) (2021)
- [Active hypothesis testing in unknown environments using recurrent neural networks and model free reinforcement learning](https://mlsanity.com/paper/2303.10623/) (2023)
- [Cooperative-Competitive Reinforcement Learning with History-Dependent Rewards](https://mlsanity.com/paper/2010.08030/) (2020)
- [Policy Evaluation in Decentralized POMDPs with Belief Sharing](https://mlsanity.com/paper/2302.04151/) (2023)
- [Graph Exploration for Effective Multi-agent Q-Learning](https://mlsanity.com/paper/2304.09547/) (2023)

## Abstract (arXiv)

This study proposes the use of a social learning method to estimate a global state within a multi-agent off-policy actor-critic algorithm for reinforcement learning (RL) operating in a partially observable environment. We assume that the network of agents operates in a fully-decentralized manner, possessing the capability to exchange variables with their immediate neighbors. The proposed design methodology is supported by an analysis demonstrating that the difference between final outcomes, obtained when the global state is fully observed versus estimated through the social learning method, is varepsilon-bounded when an appropriate number of iterations of social learning updates are implemented. Unlike many existing dec-POMDP-based RL approaches, the proposed algorithm is suitable for model-free multi-agent reinforcement learning as it does not require knowledge of a transition model. Furthermore, experimental results illustrate the efficacy of the algorithm and demonstrate its superiority over the current state-of-the-art methods.