# Intrinsic fluctuations of reinforcement learning promote cooperation

arXiv: https://arxiv.org/abs/2209.01013 · HTML: https://mlsanity.com/paper/2209.01013/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 83 references that are in the map.
- First posted on arXiv 1 Sep 2022.

## Builds on (most cited first, 18 of 18)

- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [A multi-agent reinforcement learning model of common-pool resource appropriation](https://mlsanity.com/paper/1707.06600/) (2017)
- [Maintaining cooperation in complex social dilemmas using deep reinforcement learning](https://mlsanity.com/paper/1707.01068/) (2017)
- [Deep Reinforcement Learning and its Neuroscientific Implications](https://mlsanity.com/paper/2007.03750/) (2020)
- [Statistical physics of human cooperation](https://mlsanity.com/paper/1705.07161/) (2017)
- [Evolving intrinsic motivations for altruistic behavior](https://mlsanity.com/paper/1811.05931/) (2018)
- [Consequentialist conditional cooperation in social dilemmas with imperfect information](https://mlsanity.com/paper/1710.06975/) (2017)
- [Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferences](https://mlsanity.com/paper/2011.05373/) (2020)
- [Exploration-Exploitation in Multi-Agent Learning: Catastrophe Theory Meets Game Theory](https://mlsanity.com/paper/2012.03083/) (2020)
- [Learning to Resolve Alliance Dilemmas in Many-Player Zero-Sum Games](https://mlsanity.com/paper/2003.00799/) (2020)
- [Deterministic limit of temporal difference reinforcement learning for stochastic games](https://mlsanity.com/paper/1809.07225/) (2018)
- [Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner's dilemma](https://mlsanity.com/paper/2101.11861/) (2021)
- [Artificial Intelligence & Cooperation](https://mlsanity.com/paper/2012.06034/) (2020)
- [Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability](https://mlsanity.com/paper/2109.07259/) (2021)
- [Learning dynamics explains human behavior in Prisoner's Dilemma on networks](https://mlsanity.com/paper/1401.7287/) (2014)

## Built on by (most cited first, 1 of 1)

- [How social reinforcement learning can lead to metastable polarisation and the voter model](https://mlsanity.com/paper/2406.07993/) (2024)

## Abstract (arXiv)

In this work, we ask for and answer what makes classical temporal-difference reinforcement learning with epsilon-greedy strategies cooperative. Cooperating in social dilemma situations is vital for animals, humans, and machines. While evolutionary theory revealed a range of mechanisms promoting cooperation, the conditions under which agents learn to cooperate are contested. Here, we demonstrate which and how individual elements of the multi-agent learning setting lead to cooperation. We use the iterated Prisoner's dilemma with one-period memory as a testbed. Each of the two learning agents learns a strategy that conditions the following action choices on both agents' action choices of the last round. We find that next to a high caring for future rewards, a low exploration rate, and a small learning rate, it is primarily intrinsic stochastic fluctuations of the reinforcement learning process which double the final rate of cooperation to up to 80%. Thus, inherent noise is not a necessary evil of the iterative learning process. It is a critical asset for the learning of cooperation. However, we also point out the trade-off between a high likelihood of cooperative behavior and achieving this in a reasonable amount of time. Our findings are relevant for purposefully designing cooperative algorithms and regulating undesired collusive effects.