# Evolution of Societies via Reinforcement Learning

arXiv: https://arxiv.org/abs/2410.17466 · HTML: https://mlsanity.com/paper/2410.17466/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multi-agent reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [Multi-agent reinforcement learning](https://mlsanity.com/deeprl/multi-agent-reinforcement-learning/)

## Builds on (most cited first, 15 of 15)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1906.04737/) (2019)
- [Learning to Incentivize Other Learning Agents](https://mlsanity.com/paper/2006.06051/) (2020)
- [α-Rank: Multi-Agent Evaluation by Evolution](https://mlsanity.com/paper/1903.01373/) (2019)
- [Stable Opponent Shaping in Differentiable Games](https://mlsanity.com/paper/1811.08469/) (2018)
- [DiCE: The Infinitely Differentiable Monte-Carlo Estimator](https://mlsanity.com/paper/1802.05098/) (2018)
- [Model-Free Opponent Shaping](https://mlsanity.com/paper/2205.01447/) (2022)
- [COLA: Consistent Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/2203.04098/) (2022)
- [Partner Selection for the Emergence of Cooperation in Multi-Agent Systems Using Reinforcement Learning](https://mlsanity.com/paper/1902.03185/) (2019)
- [Riemannian game dynamics](https://mlsanity.com/paper/1603.09173/) (2016)

## Abstract (arXiv)

The universe involves many independent co-learning agents as an ever-evolving part of our observed environment. Yet, in practice, Multi-Agent Reinforcement Learning (MARL) applications are typically constrained to small, homogeneous populations and remain computationally intensive. We propose a methodology that enables simulating populations of Reinforcement Learning agents at evolutionary scale. More specifically, we derive a fast, parallelizable implementation of Policy Gradient (PG) and Opponent-Learning Awareness (LOLA), tailored for evolutionary simulations where agents undergo random pairwise interactions in stateless normal-form games. We demonstrate our approach by simulating the evolution of very large populations made of heterogeneous co-learning agents, under both naive and advanced learning strategies. In our experiments, 200,000 PG or LOLA agents evolve in the classic games of Hawk-Dove, Stag-Hunt, and Rock-Paper-Scissors. Each game provides distinct insights into how populations evolve under both naive and advanced MARL rules, including compelling ways in which Opponent-Learning Awareness affects social evolution.