# Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1810.08647 · HTML: https://mlsanity.com/paper/1810.08647/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2018.

## Built on by (most cited first, 13 of 13)

- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration](https://mlsanity.com/paper/2203.08553/) (2022)
- [Quantifying the effects of environment and population diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2102.08370/) (2021)
- [Emergent Communication under Competition](https://mlsanity.com/paper/2101.10276/) (2021)
- [Neural Recursive Belief States in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.02274/) (2021)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [Fast Peer Adaptation with Context-aware Exploration](https://mlsanity.com/paper/2402.02468/) (2024)
- [InvestESG: A multi-agent reinforcement learning benchmark for studying climate investment as a social dilemma](https://mlsanity.com/paper/2411.09856/) (2024)
- [Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents](https://mlsanity.com/paper/2108.12581/) (2021)
- [Inference-Based Deterministic Messaging For Multi-Agent Communication](https://mlsanity.com/paper/2103.02150/) (2021)
- [Balancing Performance and Human Autonomy with Implicit Guidance Agent](https://mlsanity.com/paper/2109.00414/) (2021)
- [MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure](https://mlsanity.com/paper/2405.00902/) (2024)
- [Tacit Learning with Adaptive Information Selection for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.15639/) (2024)

## Abstract (arXiv)

We propose a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning (MARL), through rewarding agents for having causal influence over other agents' actions. Causal influence is assessed using counterfactual reasoning. At each timestep, an agent simulates alternate actions that it could have taken, and computes their effect on the behavior of other agents. Actions that lead to bigger changes in other agents' behavior are considered influential and are rewarded. We show that this is equivalent to rewarding agents for having high mutual information between their actions. Empirical results demonstrate that influence leads to enhanced coordination and communication in challenging social dilemma environments, dramatically increasing the learning curves of the deep RL agents, and leading to more meaningful learned communication protocols. The influence rewards for all agents can be computed in a decentralized way by enabling agents to learn a model of other agents using deep neural networks. In contrast, key previous works on emergent communication in the MARL setting were unable to learn diverse policies in a decentralized manner and had to resort to centralized training. Consequently, the influence reward opens up a window of new opportunities for research in this area.