# Multi-Agent Trust Region Policy Optimization

arXiv: https://arxiv.org/abs/2010.07916 · HTML: https://mlsanity.com/paper/2010.07916/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.09870/) (2023)
- [The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces](https://mlsanity.com/paper/2106.03352/) (2021)
- [Order Matters: Agent-by-agent Policy Optimization](https://mlsanity.com/paper/2302.06205/) (2023)
- [Coordinated Proximal Policy Optimization](https://mlsanity.com/paper/2111.04051/) (2021)
- [Heterogeneous-Agent Mirror Learning: A Continuum of Solutions to Cooperative MARL](https://mlsanity.com/paper/2208.01682/) (2022)
- [Trust Region Bounds for Decentralized PPO Under Non-stationarity](https://mlsanity.com/paper/2202.00082/) (2022)
- [A Game-Theoretic Approach to Multi-Agent Trust Region Optimization](https://mlsanity.com/paper/2106.06828/) (2021)
- [Collaborative AI Teaming in Unknown Environments via Active Goal Deduction](https://mlsanity.com/paper/2403.15341/) (2024)
- [Dealing with Non-Stationarity in MARL via Trust-Region Decomposition](https://mlsanity.com/paper/2102.10616/) (2021)
- [DM2: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching](https://mlsanity.com/paper/2206.00233/) (2022)
- [Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey](https://mlsanity.com/paper/2401.04934/) (2024)
- [Best Possible Q-Learning](https://mlsanity.com/paper/2302.01188/) (2023)
- [EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization](https://mlsanity.com/paper/2110.13373/) (2021)
- [Learning Distributed and Fair Policies for Network Load Balancing as Markov Potential Game](https://mlsanity.com/paper/2206.01451/) (2022)
- [How to Use Reinforcement Learning to Facilitate Future Electricity Market Design? Part 1: A Paradigmatic Theory](https://mlsanity.com/paper/2305.02485/) (2023)
- [A Teacher-Student MPC-PPO Coupled Reinforcement Learning Framework for Winter Temperature Control of Solar Greenhouses in Northern China](https://mlsanity.com/paper/2504.20815/) (2025)
- [Ratio-Variance Regularized Policy Optimization](https://mlsanity.com/paper/2605.26784/) (2026)
- [Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2606.25526/) (2026)
- [Reinforcement Learning: From Algorithms To Foundation Models](https://mlsanity.com/paper/2607.17560/) (2026)

## Abstract (arXiv)

We extend trust region policy optimization (TRPO) to multi-agent reinforcement learning (MARL) problems. We show that the policy update of TRPO can be transformed into a distributed consensus optimization problem for multi-agent cases. By making a series of approximations to the consensus optimization model, we propose a decentralized MARL algorithm, which we call multi-agent TRPO (MATRPO). This algorithm can optimize distributed policies based on local observations and private rewards. The agents do not need to know observations, rewards, policies or value/action-value functions of other agents. The agents only share a likelihood ratio with their neighbors during the training process. The algorithm is fully decentralized and privacy-preserving. Our experiments on two cooperative games demonstrate its robust performance on complicated MARL tasks.