# Influence-Augmented Local Simulators: A Scalable Solution for Fast Deep RL in Large Networked Systems

arXiv: https://arxiv.org/abs/2202.01534 · HTML: https://mlsanity.com/paper/2202.01534/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 56 references that are in the map.
- First posted on arXiv 3 Feb 2022.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [When to use parametric models in reinforcement learning?](https://mlsanity.com/paper/1906.05243/) (2019)
- [Control of Memory, Active Perception, and Action in Minecraft](https://mlsanity.com/paper/1605.09128/) (2016)
- [Out of Distribution Generalization in Machine Learning](https://mlsanity.com/paper/2103.02667/) (2021)
- [Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks](https://mlsanity.com/paper/2110.14300/) (2021)
- [Context-Specific Independence in Bayesian Networks](https://mlsanity.com/paper/1302.3562/) (2013)
- [Reinforcement Learning for Market Making in a Multi-agent Dealer Market](https://mlsanity.com/paper/1911.05892/) (2019)
- [Learning Compact Models for Planning with Exogenous Processes](https://mlsanity.com/paper/1909.13870/) (2019)
- [A Sufficient Statistic for Influence in Structured Multiagent Environments](https://mlsanity.com/paper/1907.09278/) (2019)
- [An Algorithm for Finding Minimum d-Separating Sets in Belief Networks](https://mlsanity.com/paper/1302.3549/) (2013)
- [Loss Bounds for Approximate Influence-Based Abstraction](https://mlsanity.com/paper/2011.01788/) (2020)
- [Influence-Augmented Online Planning for Complex Environments](https://mlsanity.com/paper/2010.11038/) (2020)
- [Offline Contextual Bandits for Wireless Network Optimization](https://mlsanity.com/paper/2111.08587/) (2021)

## Built on by (most cited first, 2 of 2)

- [Distributed Influence-Augmented Local Simulators for Parallel MARL in Large Networked Systems](https://mlsanity.com/paper/2207.00288/) (2022)
- [Perspective-Shifted Neuro-Symbolic World Models: A Framework for Socially-Aware Robot Navigation](https://mlsanity.com/paper/2503.20425/) (2025)

## Abstract (arXiv)

Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.