# Distributed Influence-Augmented Local Simulators for Parallel MARL in Large Networked Systems

arXiv: https://arxiv.org/abs/2207.00288 · HTML: https://mlsanity.com/paper/2207.00288/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 1 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity](https://mlsanity.com/paper/1707.09183/) (2017)
- [Modeling Others using Oneself in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.09640/) (2018)
- [Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.04402/) (2021)
- [A Sufficient Statistic for Influence in Structured Multiagent Environments](https://mlsanity.com/paper/1907.09278/) (2019)
- [Difference Rewards Policy Gradients](https://mlsanity.com/paper/2012.11258/) (2020)
- [Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs](https://mlsanity.com/paper/2102.10362/) (2021)
- [Dealing with Non-Stationarity in MARL via Trust-Region Decomposition](https://mlsanity.com/paper/2102.10616/) (2021)
- [Loss Bounds for Approximate Influence-Based Abstraction](https://mlsanity.com/paper/2011.01788/) (2020)
- [Influence-Augmented Online Planning for Complex Environments](https://mlsanity.com/paper/2010.11038/) (2020)
- [Influence-Augmented Local Simulators: A Scalable Solution for Fast Deep RL in Large Networked Systems](https://mlsanity.com/paper/2202.01534/) (2022)

## Abstract (arXiv)

Due to its high sample complexity, simulation is, as of today, critical for the successful application of reinforcement learning. Many real-world problems, however, exhibit overly complex dynamics, which makes their full-scale simulation computationally slow. In this paper, we show how to decompose large networked systems of many agents into multiple local components such that we can build separate simulators that run independently and in parallel. To monitor the influence that the different local components exert on one another, each of these simulators is equipped with a learned model that is periodically trained on real trajectories. Our empirical results reveal that distributing the simulation among different processes not only makes it possible to train large multi-agent systems in just a few hours but also helps mitigate the negative effects of simultaneous learning.