# Evolving intrinsic motivations for altruistic behavior

arXiv: https://arxiv.org/abs/1811.05931 · HTML: https://mlsanity.com/paper/1811.05931/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2018.

## Built on by (most cited first, 30 of 30)

- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Proximal Learning With Opponent-Learning Awareness](https://mlsanity.com/paper/2210.10125/) (2022)
- [A social path to human-like artificial intelligence](https://mlsanity.com/paper/2405.15815/) (2024)
- [Influencing Long-Term Behavior in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2203.03535/) (2022)
- [Survey on Fair Reinforcement Learning: Theory and Practice](https://mlsanity.com/paper/2205.10032/) (2022)
- [Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas](https://mlsanity.com/paper/2402.17270/) (2024)
- [Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contracts](https://mlsanity.com/paper/2407.18074/) (2024)
- [Cooperative Multi-Agent Fairness and Equivariant Policies](https://mlsanity.com/paper/2106.05727/) (2021)
- [Learning Altruistic Behaviours in Reinforcement Learning without External Rewards](https://mlsanity.com/paper/2107.09598/) (2021)
- [The Importance of Credo in Multiagent Learning](https://mlsanity.com/paper/2204.07471/) (2022)
- [Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games](https://mlsanity.com/paper/2410.07863/) (2024)
- [Exploring the Benefits of Teams in Multiagent Learning](https://mlsanity.com/paper/2205.02328/) (2022)
- [The emergence of division of labor through decentralized social sanctioning](https://mlsanity.com/paper/2208.05568/) (2022)
- [Mediated Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.08419/) (2023)
- [Similarity-based cooperative equilibrium](https://mlsanity.com/paper/2211.14468/) (2022)
- [Improved Cooperation by Exploiting a Common Signal](https://mlsanity.com/paper/2102.02304/) (2021)
- [Balancing Rational and Other-Regarding Preferences in Cooperative-Competitive Environments](https://mlsanity.com/paper/2102.12307/) (2021)
- [Artificial virtuous agents in a multiagent tragedy of the commons](https://mlsanity.com/paper/2210.02769/) (2022)
- [Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments](https://mlsanity.com/paper/2209.10958/) (2022)
- [Learning to Participate through Trading of Reward Shares](https://mlsanity.com/paper/2301.07416/) (2023)
- [Learning Roles with Emergent Social Value Orientations](https://mlsanity.com/paper/2301.13812/) (2023)
- [Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmas](https://mlsanity.com/paper/2305.00768/) (2023)
- [Incentives to Build Houses, Trade Houses, or Trade House Building Skills in Simulated Worlds under Various Governing Systems or Institutions: Comparing Multi-agent Reinforcement Learning to Generative Agent-based Model](https://mlsanity.com/paper/2411.17724/) (2024)
- [Intrinsic fluctuations of reinforcement learning promote cooperation](https://mlsanity.com/paper/2209.01013/) (2022)
- [Responsible Emergent Multi-Agent Behavior](https://mlsanity.com/paper/2311.01609/) (2023)
- [Improved cooperation by balancing exploration and exploitation in intertemporal social dilemma tasks](https://mlsanity.com/paper/2111.09152/) (2021)
- [Learning Strategic Value and Cooperation in Multi-Player Stochastic Games through Side Payments](https://mlsanity.com/paper/2303.05307/) (2023)
- [Using Protected Attributes to Consider Fairness in Multi-Agent Systems](https://mlsanity.com/paper/2410.12889/) (2024)
- [Integrated Altruistic and Fairness Preference Induces Advanced Mutual Cooperation in Sequential Social Dilemmas](https://mlsanity.com/paper/2607.04710/) (2026)

## Abstract (arXiv)

Multi-agent cooperation is an important feature of the natural world. Many tasks involve individual incentives that are misaligned with the common good, yet a wide range of organisms from bacteria to insects and humans are able to overcome their differences and collaborate. Therefore, the emergence of cooperative behavior amongst self-interested individuals is an important question for the fields of multi-agent reinforcement learning (MARL) and evolutionary theory. Here, we study a particular class of multi-agent problems called intertemporal social dilemmas (ISDs), where the conflict between the individual and the group is particularly sharp. By combining MARL with appropriately structured natural selection, we demonstrate that individual inductive biases for cooperation can be learned in a model-free way. To achieve this, we introduce an innovative modular architecture for deep reinforcement learning agents which supports multi-level selection. We present results in two challenging environments, and interpret these in the context of cultural and ecological evolution.