Dynamics of Moral Behavior in Heterogeneous Populations of Learning Agents
On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization
Abstract (arXiv)
Growing concerns about safety and alignment of AI systems highlight the importance of embedding moral capabilities in artificial agents: a promising solution is the use of learning from experience, i.e., Reinforcement Learning. In multi-agent (social) environments, complex population-level phenomena may emerge from interactions between individual learning agents. Many of the existing studies rely on simulated social dilemma environments to study the interactions of independent learning agents; however, they tend to ignore the moral heterogeneity that is likely to be present in societies of agents in practice. For example, at different points in time a single learning agent may face opponents who are consequentialist (i.e., focused on maximizing outcomes over time), norm-based (i.e., conforming to specific norms), or virtue-based (i.e., considering a combination of different virtues). The extent to which agents' co-development may be impacted by such moral heterogeneity in populations is not well understood. In this paper, we present a study of the learning dynamics of morally heterogeneous populations interacting in a social dilemma setting. Using an Iterated Prisoner's Dilemma environment with a partner selection mechanism, we investigate the extent to which the prevalence of diverse moral agents in populations affects individual agents' learning behaviors and emergent population-level outcomes. We observe several types of non-trivial interactions between pro-social and anti-social agents, and find that certain types of moral agents are able to steer selfish agents towards more cooperative behavior.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 52 references that are in the map. First posted on arXiv 7 Mar 2024.
Builds on
12 of 52 references.
References · 12
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 209
- 2018Inequity aversion improves cooperation in intertemporal social dilemmascited by 106
- 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 61
- 2020Social diversity and social preferences in mixed-motive reinforcement learningcited by 49
- 2017Consequentialist conditional cooperation in social dilemmas with imperfect informationcited by 23
- 2021Quantifying the effects of environment and population diversity in multi-agent reinforcement learningcited by 22
- 2020Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferencescited by 21
- 2019Partner Selection for the Emergence of Cooperation in Multi-Agent Systems Using Reinforcement Learningcited by 20
- 2021Mathematical foundations of moral preferencescited by 16
- 2023Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learningcited by 10
Built on by
Citing papers per year: 2024: 1.
1 paper.
- 2024Moral Alignment for LLM Agentscited by 39
- 2024Moral Alignment for LLM Agents2 Oct 2024
Similar, not cited
Close in wording, no citation either way
- 2023Understanding the World to Solve Social Dilemmas Using Multi-Agent Reinforcement Learning
- 2024Emergent Dominance Hierarchies in Reinforcement Learning Agents
- 2024Learning Fair Cooperation in Mixed-Motive Games with Indirect Reciprocity
- 2024Evolution of Societies via Reinforcement Learning
- 2026How do Role Models Shape Collective Morality? Exemplar-Driven Moral Learning in Multi-Agent Simulation
- 2025Heterogeneity in Multi-Agent Reinforcement Learning
- 2026Collective cooperation without individual fidelity in LLM agents
- 2024The impact of behavioral diversity in multi-agent reinforcement learning